feat(core): select affected tasks instead of affected projects - #36841
Draft
AgentEnder wants to merge 23 commits into
Draft
feat(core): select affected tasks instead of affected projects#36841AgentEnder wants to merge 23 commits into
AgentEnder wants to merge 23 commits into
Conversation
AgentEnder
force-pushed
the
feat/nxc-4859-affected-task-granularity
branch
from
August 28, 2026 20:01
1340f19 to
be5c9d6
Compare
Contributor
|
View your CI Pipeline Execution ↗ for commit 517a223
☁️ Nx Cloud last updated this comment at |
AgentEnder
changed the base branch from
feat/nxc-4861-files-input
to
feat/nxc-4859-task-based-affected
August 28, 2026 20:22
AgentEnder
force-pushed
the
feat/nxc-4859-affected-task-granularity
branch
from
August 29, 2026 17:06
be5c9d6 to
cf266fc
Compare
Tests the changed paths against each hash-plan instruction's globs, rather than resolving instructions to file lists and intersecting. A deleted file has no entry in the workspace file map, so a resolved list could never contain it and every rename would be missed. It also inverts the cost to O(unique instructions x changed files), and instructions are interned so a glob set shared by a thousand tasks is compiled once. TaskOutput is deliberately skipped: it resolves to a dependent task's build artifacts, which are gitignored and do not exist yet when affected runs. Dependency changes reach a consumer through task-edge propagation instead.
Builds the full candidate task graph, plans it, matches changed paths against each instruction's globs, then propagates along dependentTasksOutputFiles edges with three bits per task over a topological order. The locators still run and seed the result. A file intersection alone under-selects three ways they cover: ProjectConfiguration resolves to no files, so a project.json edit can be invisible; lockfile and external-dependency changes are not paths; and a blanket trigger like nx.json has no fileset to match. Seeding keeps this a superset of the project-grained answer. Extracts the graph marshal and the locator fan-out so filterAffected and computeAffectedTasks share one locator path rather than two. NOT yet wired to any command: planning every candidate task costs ~3ms/task on this repo, so discovering that 4 tasks are affected costs ~760ms of planning across 171 candidates. See the notes on the PR before enabling this anywhere.
Adds pruneTaskGraphToSelection, threads an optional TaskSelection through runCommand so the pruned graph reaches the runner, and dispatches on nx.json affected.granularity (NX_AFFECTED_GRANULARITY overrides). Default stays project-grained. The prune keeps each selected task's dependency closure: an affected task's upstream still has to run or restore from cache. It is applied before validation so a cycle or atomizer error names what will actually run, and re-applied after sync generators rebuild the graph. Gate only engages for `nx affected` with a target; `nx graph --affected` has no target to select against and stays project-grained.
A task can only be affected if a changed file reaches its plan, and every
route there marks the owning project: its own files, a dependency's files
inlined by ^ inputs (whose dependents the reverse walk picks up), and
{workspaceRoot} filesets, which getImplicitlyTouchedProjects attributes. So
projects-of(affected tasks) is a subset of the project-grained set, and
planning anything outside it is wasted.
Measured on this repo, selection plus hashing the selected set:
a README change 1369ms -> 6ms
a leaf e2e spec 1956ms -> 523ms
Planning every candidate to discover that nothing is affected was the whole
cost of the docs-only case, which is a common CI shape.
Also shares one HashPlanner per project graph, so the candidate pass populates
subtree_memo and instruction_pool once rather than per call.
Two target-configuration shapes create a real dependency the project graph
never records, so the reverse walk in filterAffected cannot see either:
inputs: [{ input: 'production', projects: ['shared-config'] }]
gather_project_inputs resolves the list and inlines those projects'
filesets into the task's plan, so its hash genuinely depends on them.
dependsOn: [{ projects: ['api'], target: 'build' }]
processTasksForMultipleProjects builds a task edge to a named project.
Project-grained affected under-selects for both today: a change to a project
referenced only this way selects nothing. Task granularity has to close the
hole, because it uses the project-grained answer to bound which tasks are
worth planning.
Also narrows the seeds. The file matcher is precise for ordinary source
files, so seeding every task of a touched project there just reproduced
project granularity. Seeds now cover only what glob matching cannot see:
project.json/package.json edits, lockfile changes, and the deleted-project
blanket.
Task-grained affected plans the candidate tasks to decide what is affected, and the hasher then plans the survivors. A planner carries subtree_memo, instruction_pool and external_deps_mapped across getPlans calls, so handing the same instance to both makes the second pass mostly memo hits. Threaded as an argument from computeAffectedTasks through runCommand to createTaskHasher, rather than held in a module-level cache keyed on the graph. A cache works but retains napi objects for as long as the graph is reachable; passing it explicitly scopes the lifetime to the command and avoids depending on GC timing, which a WeakRef would not. Measured on this repo, selection plus hashing the selected set, paired runs: a spec-only change 3699ms -> 3118ms a source file 3987ms -> 3647ms a project.json 3788ms -> 3260ms
Drops the nx.json affected.granularity key. This is a trial flag that we expect to remove once task granularity becomes the default, and a config key would need a migration to take back out.
I/O tracing turns an observed read of a generated artifact into a FileSetInput with includeIgnored, which can preclude an explicit dependentTasksOutputFiles input. The plan then holds a disk-backed Files instruction reading a dependency's build output with no TaskOutput anywhere in it, and propagation missed the edge entirely. dependentOutputEdges now builds consumer -> producer edges from both sources: an explicit TaskOutput, whose producer id is recovered by matching the embedded outputs, and an includeIgnored fileset whose pattern overlaps a producer's declared outputs. Matching is pattern-to-pattern against declared outputs, never the filesystem. The artifact is the thing the run would produce, so it is absent from disk and gitignored, and any check that consults disk answers "no edge" for every task that has not been built. Overlap compares literal prefixes segment-wise, so dist/libs/ui does not swallow dist/libs/ui-legacy. Producers are searched over the whole dependency closure. TaskOutput does not record whether transitive was set and a traced read cannot say how deep the producer sits, so a narrower scope could miss an edge; over-reporting costs a cache hit, missing one skips a task that needed to run. TaskOutput stays out of direct changed-file matching, since its artifacts can never appear in a diff.
The first cut walked the dependency closure per consumer, cloning task ids at every hop and re-reducing the same glob set once per task that referenced it. That was 216ms for 335 tasks here, and it scales with tasks x closure. Instructions are interned, so a glob set shared by a thousand tasks has one id and is now reduced to its literal prefixes once. The closure walk moves over positions into a flattened graph rather than hashing ids, reuses one generation-stamped scratch buffer per rayon worker, and runs in parallel. Same 1560 edges, 216ms -> 54ms. Also returns a consumer -> producers map instead of an edge list, which halves the strings crossing the boundary and is the shape the caller builds anyway.
The blanket rule fired whenever the project-grained pass happened to select every project, as a proxy for "a blanket locator returned everything". In a small workspace an ordinary source change does exactly that, so every task of every project got seeded and task granularity collapsed back to project granularity. Only a deleted project config actually needs the widening: its project is gone from the graph, so no surviving task has a fileset that names it. The check mirrors projects_from_project_glob_changes, which decides the same thing by asking whether the file is still on disk. nx.json needs no seed at all. Every plan carries it in the always-on workspace fileset (ALWAYS_ON_FILES in hash_planner), so the matcher already reaches every task precisely.
…locators The project locators and the task matcher both answer "which project owns this path", and both carried their own root map plus the same comment explaining why create_project_root_mappings cannot be used. Fixing that shared helper later now means finding one workaround rather than two. ProjectRoots and normalize_path move to affected/project_paths.rs with tests for the cases that motivated them: an empty project root, a whole-segment match so libs/a does not claim libs/a-legacy, and Windows separators. Also trims the matcher while it is being touched. Only instructions that matched a file are kept, so task selection is membership rather than collecting every hit; the per-task index vector is now built only under collectMatches, the reduce runs on rayon, and file owners borrow from the graph instead of allocating a String each. 125ms -> 42ms for 335 tasks.
…ing twice Task-grained selection plans every candidate task, then the hasher planned the surviving ones all over again. Planning costs about 3ms per task on this repo, so the second pass was roughly a second on a 335 task graph, and it is what put the task path 18% behind the project-grained one. subsetHashPlans narrows an existing HashPlans to a task list. A plan is a vector of interned instruction ids, so this filters a map and clones an Arc; the work that made the plans is shared, not repeated. It returns null when any requested task has no plan, which is the caller's signal that the plans describe some other task graph. The hasher then plans for real. That happens with an I/O snapshot bundle, whose plans are fetched after selection and so describe different hashing, and after makeAcyclic, which drops edges and changes which upstream outputs a task reads. Verified hash-identical across all 276 tasks of a real affected run, reused against freshly planned. Warm cache totals, median of three, this repo: changed task before task after project utils/path.ts 7449ms 6315ms 6291ms ab-testing.spec.ts 6510ms 5102ms 6374ms The wide case reaches parity, and the narrow case is 20% ahead because it also hashes 233 tasks rather than 278.
The env-vars-documented conformance rule fails on any NX_ variable read in source that has no row in the reference table, and NX_AFFECTED_GRANULARITY had none. That was the only violation in the run; every other rule passed.
oxlint rejects no-duplicate-imports, and both files had picked up a second import from a module they already imported: subsetHashPlans from ../native in the hasher, and TargetDependencies from ../../config/nx-json in affected.
AgentEnder
force-pushed
the
feat/nxc-4859-affected-task-granularity
branch
from
August 30, 2026 02:25
1487e21 to
aa507c2
Compare
![nx-cloud[bot]](https://avatars.githubusercontent.com/in/80458?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
…file list Three things that were carrying weight without earning it. affectedTasks returned a taskId to changed-files map behind a collectMatches flag, and nothing read either one; they were staged for the explain work. The napi boundary now returns a plain string array, and matching an instruction is a predicate rather than a list of file indices, so it short-circuits on the first hit instead of collecting every one. The lockfile set was written out by hand and was missing pnpm-lock.yml, so a pnpm workspace on the legacy name got no blanket seed and a dependency bump selected nothing. AUTO_AFFECTED_LOCK_FILES already answers exactly this and carries that name. createTaskPlanningContext took an optional pre-marshalled graph to avoid paying twice. marshalGraph is keyed on the graph, so calling it is already free the second time and the parameter is redundant.
…ed producers Resolving which upstream task's artifacts a consumer reads meant reducing every read pattern and every declared output to a literal path prefix, flattening the task graph into a positional index, and running a generation-stamped closure walk per consumer. Roughly 500 lines of rust to tell one producer from another. It does not pay. Measured on this repo across four change shapes, the precise producer set selected 2 fewer tasks out of 335 than treating a reader as reading everything it depends on, and never selected fewer than the coarse answer in any of them, so nothing was at risk of being missed: spec file, build,test,lint 74 vs 76 source file, build,test,lint 170 vs 170 project.json 119 vs 121 spec file, build 6 vs 8 Rust now reports only which tasks read upstream artifacts at all. Propagation moves to the reachability bit the task graph already supports: a task carries hasAffectedUpstream when any dependency is affected or carries it in turn, so one topological pass settles a chain in O(V+E) without materialising a closure per task, which the old walk did. The 2 extra are build tasks that were going to be cache hits. Over-reporting an edge costs a restored cache entry; missing one skips a task that needed to run.
…utputs Coarse propagation treated any task that reads an upstream artifact as reading everything it depends on, which over-selected two build tasks on this repo. Both declare dependentTasksOutputFiles naming a single file, and both sit downstream of an affected task that produces something else entirely. Recovering that needs none of the machinery it originally took. process_tasks_outputs builds one TaskOutput per dependent task from task.outputs.clone(), so the embedded vector is a producer's declared outputs and equality against task.outputs names it. No literal-prefix reduction, no overlap test, no positional index, no closure walk: the instruction only exists because the producer was already a dependency, so a lookup keyed on the output vector is enough. includeIgnored filesets stay coarse. Those name no producer at all, so a reachability bit over the dependency closure remains the only signal. Selection matches the original per-producer matching, 74 and 170 on the two comparable cases. Dropping the closure restriction only matters when two tasks declare identical outputs, which no task pair in this workspace does (177 tasks, 177 distinct output sets), and which would be one directory written twice.
The previous commit resolved dependentTasksOutputFiles precisely and left includeIgnored reads coarse, propagating those over the whole dependency closure. That was the wrong half to give up. An includeIgnored read is what I/O tracing produces once it has preempted the declared input, so it is the shape the snapshot work makes common, and this workspace has none of them today: every measurement behind the coarse decision covered only the TaskOutput path. Being coarse there had no evidence behind it. So both are resolved now, by the work each actually needs. A TaskOutput embeds its producer's declared outputs, so a lookup keyed on that vector names it, and no walk is involved. An includeIgnored read names nothing, so its patterns are compared to declared outputs on literal prefixes over the dependency closure. The walk runs only for plans carrying such a read, which is what keeps the common case free. Still much smaller than the original: the positional task index and the generation-stamped traversal are gone, and TaskOutput no longer goes through prefix reduction or overlap at all.
AgentEnder
force-pushed
the
feat/nxc-4859-affected-task-granularity
branch
from
September 1, 2026 15:58
5ce72b7 to
d8c8af6
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Each of these breaks the property the whole design rests on, that the task answer is a superset of the project one. Three are under-selection, where a task whose hash changed is not selected and silently does not run. A dependency's project config selected none of its dependents. ProjectConfiguration is spliced into a consumer's plan for each dependency and is real hash entropy, but it resolves to no files, so nothing in the consumer's filesets can match it and the seed only covered the owning project. Seeding now walks the dependents. Matching on any file the project owns would have been wrong in the other direction: it would select every consumer on a spec change, which is the case task granularity exists to narrow. A deleted project config that is not project.json or package.json selected nothing at all, where the project-grained pass it wraps returns everything. The two hardcoded basenames are replaced by the plugin globs its rust sibling already uses, so build.gradle and *.csproj count. --exclude was dropped on the task path. It is honoured in getAffectedGraphNodes, which that branch never calls, so a pipeline excluding a quarantined project started running it again. That is an explicit instruction being discarded rather than a heuristic over-selecting. The planning context survived the sync-generator graph rebuild, so tasks hashed against the pre-sync workspace. The marshalled graph and the planner built over it both describe the old graph, and @nx/js:typescript-sync moves inferred target outputs by rewriting tsconfig references. The cycle path already dropped the plans; this drops the whole context, which is what changed. Found by review of #36841.
The first fix seeded the changed project's dependents, which selects every task those projects own. For a config change on a widely depended-on project that came to 288 tasks against the project-grained answer's 278, so the case task granularity is supposed to narrow got wider instead. instruction_matches now has a ProjectConfiguration arm. The instruction is already spliced into the plan of its project and of every consumer, so matching it on the changed config resolves the same relationship precisely: a consumer is selected because it carries ProjectConfiguration for the project whose config moved, not because it happens to depend on it. Which paths count as configuration is still decided in TypeScript, where the plugins' createNodes globs are available, and passed in. A source file in the same project is not a config change and does not match, which is what keeps a spec-file change at 74 of 280 rather than widening back out.
AgentEnder
force-pushed
the
feat/nxc-4859-affected-task-granularity
branch
from
September 1, 2026 20:57
e2f5949 to
a9049ea
Compare
topologicalOrder used queue.shift(), which moves every remaining element on each pop; a read index removes the quadratic term, measured by the reviewer at 699ms over 100k ids. ROOT_TSCONFIGS duplicated ROOT_TSCONFIG_FILES in hash_planner, and its comment cited hash_tsconfig for an ordering that function does not contain. Reuses the existing const. a_deleted_file_still_matches asserted nothing about deletion: compute_affected_tasks never touches the filesystem, so the case was byte-identical to the tracked one. Renamed to what it actually pins. Also corrects the comment on the project-glob deletion probe. It claimed that leaving the path un-normalized avoids stat'ing outside the workspace, which it does not: a raw absolute path escapes just as readily, and .. escapes by ordinary resolution. The raw form is parity with the TypeScript it replaced, nothing more.
… [Self-Healing CI Rerun]
Contributor
There was a problem hiding this comment.
Nx Cloud has identified a flaky task in your failed CI:
🔂 Since the failure was identified as flaky, we triggered a CI rerun by adding an empty commit to this branch.
🔔 Heads up, your workspace has pending recommendations ↗ to auto-apply fixes for similar failures.
🎓 Learn more about Self-Healing CI on nx.dev
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current Behavior
nx affectedselects projects, so every target of every dependent runs whether or not the change can reach it. Changing a spec file inpackages/nxselects all 45testtasks here, including projects whose test inputs never see that file.Two target configs also create a dependency the project graph never records:
{ "inputs": [{ "input": "production", "projects": ["shared-config"] }], "dependsOn": [{ "projects": ["api"], "target": "build" }], }gather_project_inputsinlines the named project's filesets into the consumer's plan, andprocessTasksForMultipleProjectsbuilds a task edge to it. Neither needs a project-graph edge, so the reverse walk misses both, and a change to a project referenced only this way selects nothing.Expected Behavior
nx affectedselects tasks whose inputs the change reaches. We build the candidate task graph, resolve each task's hash plan, and match the changed paths against each instruction's globs.Off by default, opt in with
NX_AFFECTED_GRANULARITY=task. Env var only, so the trial leaves no config to migrate away later.Selection
Nx repo, changing one spec file. Selection marks 74 of 280 candidate tasks: 65
lint, 2test, 1build, and nothing inbuild-baseortypecheck.-t test, test tasks-t test, whole graph-t build-t build,test,lintThe first row is the win. The rest of each graph is the dependency closure of the selected tasks, which has to stay so the selected ones can run; those restore from cache.
lintreads nearly every file in a project, so a graph dominated bylintre-inflates and-t build,test,lintbarely moves.Cost
Selection now plans the candidate tasks, which the hasher then reuses instead of planning again, so a run plans once rather than twice.
show projects --affectedaffected -t lint, fully cachedTask hashes are unchanged: this branch, master and nx 23.1.2 all hit the same cache entries.
show projects --affectedis still project-grained, so there is no task column for it.Related Issue(s)
NXC-4859