Bazel support

[brandonchinn178]

Describe the Bug

I'm evaluating Pyrefly for my company's Bazel project because mypy is too slow with caching disabled. However, we're seeing even worse performance from Pyrefly - over 2x slowdown.

Notable things with our Bazel setup:

• Each dependency gets installed into a separate directory
  • When running a Python file, Bazel simply adds all the dependencies to PYTHONPATH. With pyrefly, I add these directories to --site-package-path
• Each Python file is a Bazel target, and each Bazel target is typechecked independently
  • With mypy, we have follow_imports set to silent
  • If pyrefly is optimized for checking a whole project at once, it might not be good for multiple runs checking a single file?

Sandbox Link

No response

(Only applicable for extension issues) IDE Information

No response

[yangdanny97]

Hi, would you mind sharing a bit more info about the mypy & pyrefly config settings you're using?

Without more details off the top of my head one thing to look at might be https://pyrefly.org/en/docs/configuration/#untyped-def-behavior

Mypy's behavior is the equivalent of our "skip-and-infer-return-any"

[yangdanny97]

If pyrefly is optimized for checking a whole project at once, it might not be good for multiple runs checking a single file?

We do support incremental checking, it's key for LSP performance as files are being edited. For typechecking specifically, I think running pyrefly check is always a full check since state isn't preserved between runs. I think there is a watch mode, but IDK how good it is.

@connernilsen can comment more on this, he is working on our integration with the BUCK build system

[brandonchinn178]

We do support incremental checking, it's key for LSP performance as files are being edited. For typechecking specifically, I think running pyrefly check is always a full check since state isn't preserved between runs. I think there is a watch mode, but IDK how good it is.

Right, sorry, I meant multiple runs checking a single file each. I'm thinking maybe pyrefly has optimized this situation:

pyrefly check file1.py file2.py file3.py

and maybe not optimized this situation (e.g. maybe file2.py imports file1.py):

pyrefly check file1.py & pyrefly check file2.py & pyrefly check file3.py

Hi, would you mind sharing a bit more info about the mypy & pyrefly config settings you're using?

The above experiment was done with this command:

pyrefly check \
  --disable-search-path-heuristics true \
  --search-path . \
  --search-path bazel-out/darwin_arm64-fastbuild/bin/ \
  --site-package-path bazel-out/external/dep1/site-packages \
  --site-package-path bazel-out/external/dep2/site-packages \
  --python-version 3.11 \
  file1.py

And these mypy settings:

strict = true
explicit_package_bases = true
check_untyped_defs = true
pretty = true

# Note: it seems like pyrefly doesn't support this?
warn_unreachable = true

# suppress type-checking third-party modules
# https://github.com/theoremlp/rules_mypy#customizing-mypy
follow_imports = silent
follow_imports_for_stubs = true

# Note: We set `strict = true` above, but the files I'm checking
# in this experiment aren't fixed with strict yet, so we set
# these overrides to unset `strict = true`
# (strict = false doesn't work in [[tool.mypy.overrides]])
disallow_any_generics = false
disallow_subclassing_any = false
disallow_untyped_calls = false
disallow_untyped_defs = false
disallow_incomplete_defs = false
disallow_untyped_decorators = false
warn_return_any = false
no_implicit_reexport = false
strict_equality = false
extra_checks = false

So I think the setting we want is --untyped-def-behavior=check-and-infer-return-any. But adding that flag doesn't improve the runtime

[samwgoldman]

I'm thinking maybe pyrefly has optimized this situation [...] and maybe not optimized this situation [...]

I think this is right. I think #360 is related here, since Pyrefly will eagerly check everything reachable from the file you check, which would definitely go quadratic in cases like this.

I'm less familiar with Bazel compared to Buck2, but I think we will end up solving a lot of the same problems either way. Ideally, we would only need to look at a given target once, and from looking at that target understand its typed interface, then check other targets w.r.t. that typed interface. This kind of optimization is particularly important because of return type inference, which Pyrefly has but Mypy doesn't.

As a smaller step, just making sure that pyrefly check file2.py is lazier when getting types from file1.py would help. That's where a solution to #360 might help.

[connernilsen]

Hey @brandonchinn178, super excited to hear about your interest in build system support!

It's still very experimental, but I just put up PRs for adding in custom build system queries. Things may not work super well yet, but if set up correctly, you should be able to do non-watch type checks and run an LSP.

#1243 has the changes for that, but you'll have to supply your own build-system specific script for now.

Your config will look something like this:

skip-interpreter-query = true
disable-search-path-heuristics = true

[build-system]
type = "custom"
command = ["your", "command", "to", "run"]

It expects to be able to pass an argfile to the command as the only argument with the structure:

--
--file
<file1>
--file
<file2>
...

and expects the following structure back:

{
  "db": {
    "<target>": {
      "deps": [
        "<dependency target name>",
        ...
      ],
      "srcs": {
        "<dotted module name (for import lookup)>": [
          "<any py or pyi files that should be found here for this target>",
          ...
        ],
        "python_version": "3.<minor version>",
        "python_platform": "<python platform>",
        "buildfile_path": "<the buildfile defining the rule producing this target>"
      }
    },
    ...
  },
  "root": "<path to prefix onto the returned files>"
}

Again, this is very experimental, so expect bugs and possible API changes if you want to try it out, but hopefully I'll be able to get it more production-ready in the next few weeks and provide real documentation.

[brandonchinn178]

I actually had the same idea for our mypy support 😄

Previously, I was evaluating both mypy and pyrefly with Bazel using one-off CLI invocations, but I just learned about Bazel's persistent worker API, and I'm currently implementing it for mypy. It's basically a similar system to what you have here.

Fingers crossed, I should see major speedups on the mypy side, and then maybe I can try pyrefly again with this technique, and hopefully see even more speed up!

[brandonchinn178]

hm actually if I'm reading your PR correctly, I actually want the other direction; I'd want a Python script that can do something like:

run("pyrefly server start")
try:
    while True:
        req = parse_req(stdin.readline())
        resp = run("pyrefly server send", stdin=req)
        stdout.write(resp)
finally:
    run("pyrefly server stop")

Would this be possible?