perf: content-hash based cache keys for CI compatibility by zzzz465 · Pull Request #5 · goforj/wire

zzzz465 · 2026-03-27T10:39:29Z

Replace mtime-based cache invalidation with content hashing:

artifact_cache: use SHA-256 of file content instead of ModTime
discovery_cache: use content hash for file matching, add WIRE_DISCOVERY_CACHE_DIR env var
Bump cache versions (artifact v4, discovery v4)

This enables wire cache to work correctly in CI environments where file mtimes are not preserved across runs (e.g., S3 cache restore, git checkout).

Replace mtime-based cache invalidation with content hashing: - artifact_cache: use SHA-256 of file content instead of ModTime - discovery_cache: use content hash for file matching, add WIRE_DISCOVERY_CACHE_DIR env var - Bump cache versions (artifact v4, discovery v4) This enables wire cache to work correctly in CI environments where file mtimes are not preserved across runs (e.g., S3 cache restore, git checkout). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

cmilesio · 2026-03-29T20:16:33Z

Hey! Thanks for contributing.

Main concern for this would be overhead introduced from doing content based hashing. Ran the benchmarks locally and didn't seem to make a substantial difference to cause any concern.

There's a few cache knobs that I think we should unify base env vars on but I'm totally fine with that change as well.

I'm glad the new loader changes has made quite a bit of difference for you. I imagine this was on some pubg services?

scripts/import-benchmarks.sh scenarios

+---------------+-------+--------+----------+----------------------+----------+----------+---------+
|       profile | local | stdlib | external | change type          | stock    | current  | speedup |
+---------------+-------+--------+----------+----------------------+----------+----------+---------+
|         local | 41    | 191    | 1        | cold run             | 379.0ms  | 3088.8ms | 0.12x   |
|         local | 41    | 191    | 1        | unchanged rerun      | 341.8ms  | 9.1ms    | 37.65x  |
|         local | 41    | 191    | 1        | body-only local edit | 325.5ms  | 36.8ms   | 8.85x   |
|         local | 41    | 191    | 1        | shape change         | 324.8ms  | 149.3ms  | 2.18x   |
|         local | 41    | 191    | 1        | import change        | 359.7ms  | 149.0ms  | 2.41x   |
|         local | 41    | 191    | 1        | known import toggle  | 326.4ms  | 149.6ms  | 2.18x   |
|    local-high | 1016  | 191    | 1        | cold run             | 741.9ms  | 5134.3ms | 0.14x   |
|    local-high | 1016  | 191    | 1        | unchanged rerun      | 603.9ms  | 81.6ms   | 7.40x   |
|    local-high | 1016  | 191    | 1        | body-only local edit | 599.5ms  | 173.5ms  | 3.45x   |
|    local-high | 1016  | 191    | 1        | shape change         | 587.3ms  | 690.3ms  | 0.85x   |
|    local-high | 1016  | 191    | 1        | import change        | 591.0ms  | 696.3ms  | 0.85x   |
|    local-high | 1016  | 191    | 1        | known import toggle  | 610.3ms  | 677.8ms  | 0.90x   |
|  external-low | 42    | 243    | 342      | cold run             | 1588.3ms | 8043.6ms | 0.20x   |
|  external-low | 42    | 243    | 342      | unchanged rerun      | 1216.7ms | 17.6ms   | 69.05x  |
|  external-low | 42    | 243    | 342      | body-only local edit | 1172.3ms | 115.0ms  | 10.19x  |
|  external-low | 42    | 243    | 342      | shape change         | 1233.5ms | 421.0ms  | 2.93x   |
|  external-low | 42    | 243    | 342      | import change        | 1253.9ms | 426.1ms  | 2.94x   |
|  external-low | 42    | 243    | 342      | known import toggle  | 1200.1ms | 412.3ms  | 2.91x   |
| external-high | 117   | 243    | 342      | cold run             | 1565.6ms | 8419.3ms | 0.19x   |
| external-high | 117   | 243    | 342      | unchanged rerun      | 1302.9ms | 22.6ms   | 57.74x  |
| external-high | 117   | 243    | 342      | body-only local edit | 1173.0ms | 125.5ms  | 9.34x   |
| external-high | 117   | 243    | 342      | shape change         | 1247.7ms | 460.6ms  | 2.71x   |
| external-high | 117   | 243    | 342      | import change        | 1100.0ms | 464.9ms  | 2.37x   |
| external-high | 117   | 243    | 342      | known import toggle  | 1172.0ms | 465.0ms  | 2.52x   |
+---------------+-------+--------+----------+----------------------+----------+----------+---------+

Replace mtime-based cache invalidation with content hashing: - artifact_cache: use SHA-256 of file content instead of ModTime - discovery_cache: use content hash for file matching, add WIRE_DISCOVERY_CACHE_DIR env var - Bump cache versions (artifact v4, discovery v4) This enables wire cache to work correctly in CI environments where file mtimes are not preserved across runs (e.g., S3 cache restore, git checkout). Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: incremental loading * feat(incremental): reuse unchanged local packages in fast path and harden fallback behavior * perf(incremental): trim cold bootstrap work and keep warm shape changes fast * perf(incremental): load deps conditionally * chore(incremental): clear session cache * fix(cli): improve wire error coloring and solve error labeling * feat(incremental): harden loader and scenario tooling * feat: custom loader initial * feat: external loader caching * chore: remove local caching strat * feat: local caching from wire perspective * feat: go dep cache * feat(loader): cache unchanged root output * chore: bench tweaks * chore: re-implement cache * fix: provider discovery * chore: benchmark update * fix: ci * fix: ci * fix: windows tmpdir issue * fix: windows bench executable path * fix(loader): strengthen artifact keys for replaced external modules * test(loader): harden cache invalidation and discovery parity coverage * fix(loader): treat replaced workspace deps as local and harden runtests * fix(loader): make cache-hardening tests and runtests portable * fix(loader): use valid file GOPROXY URLs in proxy-based tests * fix(loader): format file GOPROXY URLs correctly on windows * fix(loader): normalize test path comparisons across platforms * refactor: remove unused loader and wire helpers * refactor: dedupe command and custom loader helpers * refactor: make loader artifact policy explicit * refactor: dedupe custom loader import linking * refactor: share import target resolution in custom loader * refactor: centralize types info setup in custom loader * refactor: share parse error conversion in custom loader * refactor: share source parsing in custom loader * refactor: centralize semantic artifact cache inputs * refactor: isolate semantic artifact cache io * refactor: isolate semantic provider set artifact lookup * refactor: extract semantic provider set item application * refactor: share semantic struct field helpers * refactor: share semantic package object lookup * refactor: share semantic output type assembly * refactor: share struct provider shell assembly * refactor: share allowed struct field inputs * refactor: share selected struct field inputs * refactor: share field output assembly for FieldsOf * refactor: share quoted struct field lookup * refactor: share semantic pointer expansion * refactor: reuse field parent struct resolution * refactor: share field object assembly * refactor: share named struct type resolution * refactor: share semantic type name lookup * refactor: share semantic package member lookup * refactor: share semantic error wrapping * refactor: share provider set finalization * refactor: add isolated output cache gate * refactor: make provider set fallback policy explicit * refactor: share custom loader root loading path * refactor: share custom loader metadata root graph * refactor: isolate semantic provider set support rule * refactor: fold back weak cleanup abstractions * refactor: narrow loader semantic artifact coupling * refactor: unify semantic provider set support rules * fix: restore local loader artifact safety gate * refactor: disable semantic reconstruction by default * refactor: remove semantic reconstruction path * refactor: remove semantic cache layer * refactor: add import benchmark profile filter * refactor: trim redundant discovery cache metadata * refactor: remove redundant discovery cache cloning * refactor: split external benchmark profiles * refactor: share custom typed load pipeline * refactor: centralize custom metadata loading * refactor: share loader fallback reason policy * refactor: add targeted local profile benchmark * refactor: add one-shot import profile harness * perf: reuse root discovery for generate loads * style: format loader discovery changes * perf: content-hash based cache keys for CI compatibility (#5) Replace mtime-based cache invalidation with content hashing: - artifact_cache: use SHA-256 of file content instead of ModTime - discovery_cache: use content hash for file matching, add WIRE_DISCOVERY_CACHE_DIR env var - Bump cache versions (artifact v4, discovery v4) This enables wire cache to work correctly in CI environments where file mtimes are not preserved across runs (e.g., S3 cache restore, git checkout). Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: share wire cache base directory * docs: add readme in internal for future travelers * feat: support method expression providers (#6) * feat(caching): content based caching to discovery, loader, output --------- Co-authored-by: jungooji <zzzz465@naver.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: incremental loading * feat(incremental): reuse unchanged local packages in fast path and harden fallback behavior * perf(incremental): trim cold bootstrap work and keep warm shape changes fast * perf(incremental): load deps conditionally * chore(incremental): clear session cache * fix(cli): improve wire error coloring and solve error labeling * feat(incremental): harden loader and scenario tooling * feat: custom loader initial * feat: external loader caching * chore: remove local caching strat * feat: local caching from wire perspective * feat: go dep cache * feat(loader): cache unchanged root output * chore: bench tweaks * chore: re-implement cache * fix: provider discovery * chore: benchmark update * fix: ci * fix: ci * fix: windows tmpdir issue * fix: windows bench executable path * fix(loader): strengthen artifact keys for replaced external modules * test(loader): harden cache invalidation and discovery parity coverage * fix(loader): treat replaced workspace deps as local and harden runtests * fix(loader): make cache-hardening tests and runtests portable * fix(loader): use valid file GOPROXY URLs in proxy-based tests * fix(loader): format file GOPROXY URLs correctly on windows * fix(loader): normalize test path comparisons across platforms * refactor: remove unused loader and wire helpers * refactor: dedupe command and custom loader helpers * refactor: make loader artifact policy explicit * refactor: dedupe custom loader import linking * refactor: share import target resolution in custom loader * refactor: centralize types info setup in custom loader * refactor: share parse error conversion in custom loader * refactor: share source parsing in custom loader * refactor: centralize semantic artifact cache inputs * refactor: isolate semantic artifact cache io * refactor: isolate semantic provider set artifact lookup * refactor: extract semantic provider set item application * refactor: share semantic struct field helpers * refactor: share semantic package object lookup * refactor: share semantic output type assembly * refactor: share struct provider shell assembly * refactor: share allowed struct field inputs * refactor: share selected struct field inputs * refactor: share field output assembly for FieldsOf * refactor: share quoted struct field lookup * refactor: share semantic pointer expansion * refactor: reuse field parent struct resolution * refactor: share field object assembly * refactor: share named struct type resolution * refactor: share semantic type name lookup * refactor: share semantic package member lookup * refactor: share semantic error wrapping * refactor: share provider set finalization * refactor: add isolated output cache gate * refactor: make provider set fallback policy explicit * refactor: share custom loader root loading path * refactor: share custom loader metadata root graph * refactor: isolate semantic provider set support rule * refactor: fold back weak cleanup abstractions * refactor: narrow loader semantic artifact coupling * refactor: unify semantic provider set support rules * fix: restore local loader artifact safety gate * refactor: disable semantic reconstruction by default * refactor: remove semantic reconstruction path * refactor: remove semantic cache layer * refactor: add import benchmark profile filter * refactor: trim redundant discovery cache metadata * refactor: remove redundant discovery cache cloning * refactor: split external benchmark profiles * refactor: share custom typed load pipeline * refactor: centralize custom metadata loading * refactor: share loader fallback reason policy * refactor: add targeted local profile benchmark * refactor: add one-shot import profile harness * perf: reuse root discovery for generate loads * style: format loader discovery changes * perf: content-hash based cache keys for CI compatibility (#5) Replace mtime-based cache invalidation with content hashing: - artifact_cache: use SHA-256 of file content instead of ModTime - discovery_cache: use content hash for file matching, add WIRE_DISCOVERY_CACHE_DIR env var - Bump cache versions (artifact v4, discovery v4) This enables wire cache to work correctly in CI environments where file mtimes are not preserved across runs (e.g., S3 cache restore, git checkout). * refactor: share wire cache base directory * docs: add readme in internal for future travelers * feat: support method expression providers (#6) * feat(caching): content based caching to discovery, loader, output --------- Co-authored-by: jungooji <zzzz465@naver.com>

cmilesio approved these changes Mar 29, 2026

View reviewed changes

cmilesio merged commit b0177a6 into goforj:cmilesdev/custom-loader Mar 29, 2026
4 checks passed

zzzz465 mentioned this pull request Apr 2, 2026

fix: output_cache.go is still using mtime instead of hashing #7

Closed

zzzz465 deleted the fix/artifact-cache-content-hash branch April 2, 2026 01:27

zzzz465 mentioned this pull request Apr 2, 2026

perf: custom loader (up to 70x+ improvements) #3

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: content-hash based cache keys for CI compatibility#5

perf: content-hash based cache keys for CI compatibility#5
cmilesio merged 1 commit into
goforj:cmilesdev/custom-loaderfrom
zzzz465:fix/artifact-cache-content-hash

zzzz465 commented Mar 27, 2026

Uh oh!

cmilesio commented Mar 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

zzzz465 commented Mar 27, 2026

Uh oh!

cmilesio commented Mar 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants