Incremental compilation

[mlugg]

How do I try it?

TL;DR: zig build --watch -fincremental --prominent-compile-errors, either targeting x86_64-linux with the self-hosted backend, or just not emitting a binary (i.e. -fno-emit-bin).

The easiest way to try out incremental compilation is passing the --watch -fincremental flags to zig build: the build system will track when source files change and perform incremental updates. (Note that --watch does have bugs on macOS right now; #21905). It might also be helpful to add the --prominent-compile-errors flag.

The self-hosted x86_64 backend targeting ELF and the C backend have fairly good support for incremental compilation. Other backends are almost certain to crash or miscompile at the moment. However, a common use case for incremental compilation is for quick compile error feedback when trying to get code compiling, in which case you might want to consider emitting no binary at all (in the build system, don't install the compile step artifact, just depend on the compile step directly).

Using incremental compilation makes you more likely to encounter compiler bugs, including false positive compile errors, false negative compile errors, miscompilations, and more. If you encounter one and are able to figure out how to consistently reproduce it, bug reports are appreciated.

---

Rather than bugs in existing parts of the incremental compilation logic, this issue tracks all logic which is explicitly WIP or incomplete, such as most linkers.

Codegen and linking

• self-hosted ELF linker
  • Solved with the merge of Elf2: create a new linker from scratch #25299. Although the new linker is currently missing many features, it is the default when using -fincremental, and has no known issues with incremental compilation.
• self-hosted COFF linker
  • Coff2: create a new linker from scratch #25430 made progress towards this, but @jacobly0 reports that we're sporadically affected by some kind of race condition in the Windows kernel, meaning this linker is currently buggy. We can check this off once that's solved.
• self-hosted MachO linker
• LLVM backend
  • this is low priority as there isn't much to gain from it; we still have to wait for the full LLVM emit every update

Other

• incremental updates for std.builtin.panic and other cached values
  • some values are cached in Zcu, e.g. Zcu.panic_func_index
  • so, updates to panic don't register the correct dependencies and can crash the compiler or worse
• incremental updates for std.builtin types
  • we currently don't register dependencies when doing this; Zcu.PerThread.getBuiltin etc should live on Sema so as to register appropriate dependencies
• @embedFile (incremental: handle @embedFile #22602)
• enable comptime call memoization
  • first, do some experiments to see how valuable memoization actually is confirmed: sometimes very important for compilation speed!
  • associated each memoized call with an AnalUnit to track dependencies properly
• figure out how to handle test references
  • determining which tests are referenced at the end of an update would require a resolveReferences
  • if we're doing that, let's just memoize its result in a field on Zcu
  • but, changing the test discovery rules could be an alternative solution, since it's an idea that's been floated anyway
• for efficiency, memoize the result of Zcu.resolveReferences in a field on Zcu to avoid recomputing it when we call getAllErrorsAlloc/totalErrorCount multiple times

Tooling

• fuzzing infrastructure to find crashes and incorrect behavior in incremental compilation
• debug tooling to dump dependency information into a graphviz/similar file
• a compact binary format giving information about invalidated dependencies
  • if the compiler generates this under -fincremental and has a way to give it to the user, this could be useful for incremental bug reports

[andrewrk]

Regarding fuzzing, here are some properties we can test in addition to the obvious ones (i.e. not panicking or crashing)

• path independence: regardless of the incremental steps taken along the way, two compilations based on the same set of input files should result in the same output binary and/or compilation errors.
• Detection of triggering too much invalidation: test cases can be generated in an environment that includes the standard library; no matter what test case is generated, it should not cause, for example, std.process.exit to be reanalyzed. This applies to everything not generated by the test case, assuming the test case cannot set std options.

[mlugg]

Detection of triggering too much invalidation

It's worth noting that parts of std/start.zig can correctly be invalidated if root.main (or root._start etc) changes.

EDIT: so can a few other things, like std.options and std.builtin.panic.

[mlugg]

@reader-lamp no; building in ReleaseSafe will not be subject to the performance improvements of incremental compilation (at least not by default). Release builds are allowed, and even expected, to take several orders of magnitude longer to compile than Debug. If you want fast compilation speed, that's what Debug is for, and that's where incremental compilation can bring us significant gains.