Replace failing ty with Pyrefly

[abitrolly]

Purpose

Fix CI/CD failures in PRs and main branch.

References

Fixes #14238

• <...>
• <...>
• <...>

AI Disclosure

No AI tools used

[yangdanny97]

I don't think we should point it at main, that would cause too many issues because Pyrefly is actively developed and CI would change day to day. Should change pyproject.toml to pin Pyrefly to v1.1.0 (right now it's unpinned) and mirror the same setup as ty here - so no need to delete the uv stuff & installing dependencies.

I would suggest the PR be limited to adding Pyrefly + resolving any issues Pyrefly flags, rather than deleting the ty config, to avoid doing too much.

[abitrolly]

@yangdanny97 then instructions at https://pyrefly.org/en/docs/installation/#using-the-github-action-recommended should somehow reflect the latest stable version. Right now they point to @main.

[abitrolly]

Should change pyproject.toml to pin Pyrefly to v1.1.0 (right now it's unpinned)

@yangdanny97 I pinned action to 1.1.0, not to v1.1.0, because the latter failed with Error: Unable to resolve action `facebook/pyrefly@v1.1.0`, unable to find version `v1.1.0`

Still the same error with pinned version.

Download action repository 'actions/checkout@v6' (SHA:df4cb1c069e1874edd31b4311f1884172cec0e10)
Download action repository 'facebook/pyrefly@1.1.0' (SHA:67a11d42e5b1f141e3aa5340387da4f94292926d)
Error: Could not find file '/home/runner/work/_actions/_temp_a2996619-fddd-427e-acdf-6d1ebded008a/_staging/pyrefly-67a11d42e5b1f141e3aa5340387da4f94292926d/pyrefly_wasm/rust-toolchain'.

I am not sure why pyproject.toml should be changed if I use action as described in https://pyrefly.org/en/docs/installation/#using-the-github-action-recommended

[yangdanny97]

yeah I think maybe ignore the instructions on the website, I need to fix that.

what i mean is:

• change https://github.com/sphinx-doc/sphinx/blob/master/pyproject.toml#L131 to pin pyrefly to 1.1.0
• the github action workflow for pyrefly should look basically the exact same as ty's except it does pyrefly check instead of ty check

[abitrolly]

@yangdanny97 if pyrefly needs to be injected into project dependencies, then what is the point in action?

[yangdanny97]

@abitrolly I don't understand what you mean - the type checkers and type stubs are listed in their own dependency group, so I don't think they're downloaded when a regular user downloads the package.

Listing it like this allows developers to run the correct version of the type checker locally, without needing to submit to Github and waiting for CI (and if CI reads the version from the same place then it's easy to keep them both in sync)

[abitrolly]

@yangdanny97 the point is to replace failing ty checker (that can not fixed) with something that doesn't bring CI down.

[picnixz]

I don't understand why we are actually having 3 different type checkers. I know that mypy is slow but for our needs, mypy has been used for a very long time. And pyrefly is still in its first minor bugfix and I don't know whether it's also that stable.

If we want to ditch ty for pyrefly, pyrefly should be declared a dependency not in the CI but for local dev as well. It doesn't make sense otherwise to locally use ty but not for CI checks.

[yangdanny97]

Pyrefly is fairly stable now, given that it's past v1.0. There is a migration script that can handle changes to suppressed errors and whatnot during version bumps.

Re: multiple type checkers, I was under the impression that outside of custom extensions sphinx is mostly called from the CLI rather than used as a library, is that accurate?

[picnixz]

Re: multiple type checkers, I was under the impression that outside of custom extensions sphinx is mostly called from the CLI rather than used as a library, is that accurate?

Yes, AFAIK usually you use Sphinx but you don't write extensions for it. However, I still fail to see why sphinx itself should have multiple type checkers.

[yangdanny97]

The main benefit of running multiple type checkers for library maintainers is that they can verify the common usage patterns are compatible with their users' tooling.

The usual guidance we'd give is to pick 1 type checker to run on the source code of the library, and only run multiple type checkers on the code that represents what the users would write.

For most libraries this would be the test suite, but in this case I guess it would be any sample/built-in extensions that are in this repo.

Whether or not it's worth the hassle to maintain this setup if only a small fraction of users write extensions is something that you can decide for yourselves.

I personally think running a single type checker for the whole project is perfectly fine in this case, and additional checkers only adds marginal benefits.

That said, the Pyrefly team (of which I am a member) is happy to provide support & bug fixes if you do choose to use Pyrefly, and there are benefits like having a faster tool that has better IDE integration than mypy.

[timhoffm]

Generally, I‘d be in favor of switching to pyrefly. It seems currently the best solution, and the lock-in effect is weak. We can always change again later.

[picnixz]

According to pip, pyrefly 1.1.1 should be available through pip, so can you pin that in pyproject.toml first and use that one for the CI instead of the GHA? I don't know whether it's better if we use GHA or uv here though but since the CI uses pip everywhere, I'd prefer keeping the same structure for now (and later we can switch to GHA for all actions).

By pinning somethjing in pyproject.toml we also ensure that we don't have divergent results when checking locally.

And if we drop ty, I would also remove it from pyproject.toml.