fix(compile): preserve nested worktree files

[danielmeppiel]

fix(compile): preserve nested worktree files

TL;DR

apm compile --clean now treats a nested Git worktree as a cleanup boundary, so it cannot delete that checkout's tracked APM-generated AGENTS.md files. The orphan scanner still removes stale files in the parent project. This closes #2436 with a real-CLI lifecycle regression test, a mutation-proof architecture guard, and a documented --clean boundary.

Important

Closes #2436. The lifecycle proof commits AGENTS.md in a nested worktree, runs compile --clean from its parent, and verifies both bytes and git status --porcelain.

Problem (WHY)

• Issue #2436 reports that compile --clean deleted tracked AGENTS.md files belonging to worktrees nested below the project root.
• The old recursive Path.rglob("AGENTS.md") scan crossed checkout boundaries, so cleanup treated an independently tracked file as a parent orphan.
• [!] The failure was silent until a user inspected the nested checkout's Git status.

Why these matter: cleanup must honor a checkout boundary before it decides that a generated file is disposable. The regression proof follows the repository principle that "Grounding outputs in deterministic tool execution transforms probabilistic generation into verifiable action."

Approach (WHAT)

#	Fix
1	Replace the unprunable recursive glob with os.walk, removing excluded directories before descent.
2	When a non-root directory contains a .git file, clear its child directory list and skip the whole nested worktree.
3	Preserve the marker gate and parent-orphan cleanup path; prove the boundary through the installed CLI.

Implementation (HOW)

• src/apm_cli/compilation/distributed_compiler.py -- walks candidate directories so it can prune linked-worktree roots before considering AGENTS.md; the top-level worktree remains eligible.
• tests/integration/test_compile_clean_nested_worktree.py -- creates a real nested Git worktree, commits its marked AGENTS.md, runs compile --clean, and checks parent cleanup plus nested byte and status invariants.
• scripts/lint-architecture-boundaries.sh and tests/integration/test_architecture_authorities.py -- require the prunable walk and .git-file boundary; the architecture test mutates the predicate and expects the guard to fail.
• docs/src/content/docs/reference/cli/compile.md -- states that --clean excludes nested Git worktrees.

Diagrams

Legend: the dashed node is the new boundary that prevents parent cleanup from entering a linked worktree.

flowchart LR
    subgraph Parent[Parent project]
        Scan[os.walk cleanup scan]
        Orphan[stale AGENTS.md]
    end
    subgraph Nested[Nested Git worktree]
        Boundary[.git file boundary]
        Tracked[tracked AGENTS.md]
    end
    Scan --> Orphan
    Scan --> Boundary
    Boundary -. prunes descendants .-> Tracked
    classDef new stroke-dasharray: 5 5;
    class Boundary new;

Loading

Trade-offs

• Prunable walk over rglob. Chose os.walk because its mutable child list stops descent; rejected post-filtering because it still scans the nested checkout.
• .git file boundary. Chose Git's linked-worktree marker; rejected an allowlist such as .worktrees because nested worktrees can use any directory name.
• Lifecycle plus static proof. Chose a real CLI/Git scenario for user-visible behavior and a narrow source-shape guard for future regressions; the guard intentionally protects this scanner rather than general Git discovery.

Benefits

1. A parent compile --clean removes the fixture's stale parent orphan while preserving one tracked nested AGENTS.md.
2. The nested checkout remains byte-identical and returns empty porcelain status after cleanup.
3. A predicate-removal mutation fails the lifecycle regression test, and the architecture guard independently rejects the missing boundary.

Validation

APM_E2E_TESTS=1 uv run pytest -q tests/integration/test_compile_clean_nested_worktree.py:

.                                                                        [100%]
1 passed in 1.74s

uv run pytest -q tests/unit/compilation/test_distributed_compiler_hermetic.py:

.....................................................................    [100%]
69 passed in 0.57s

Repository quality and lint evidence

uv run --extra dev pytest -p no:cacheprovider -q tests/quality:

......................................................                   [100%]
54 passed in 80.24s (0:01:20)

uv run --frozen python scripts/check_test_assertions.py && uv run --frozen python scripts/check_exact_test_duplicates.py:

[+] assertion-quality ratchet clean: AQ001=4, AQ002=12
[+] exact test duplicate ratchet clean: 1095 files, 0 allowed duplicate group(s)

CI lint mirror:

All checks passed!
1594 files already formatted

Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

[+] auth-signal lint clean
[+] architecture boundary lint clean

Scenario Evidence

#	Scenario (user promise)	Principle(s)	Test(s) proving it	Type
1	Run apm compile --clean in a project with a nested Git worktree: stale parent output is removed, while the nested checkout keeps its tracked instructions and stays clean.	Governed by policy, DevX (pragmatic as npm)	tests/integration/test_compile_clean_nested_worktree.py::test_compile_clean_preserves_nested_git_worktree_agents_file (regression-trap for #2436)	e2e
2	Remove the nested-worktree boundary: the lifecycle regression test fails instead of silently accepting deletion.	Governed by policy	tests/integration/test_compile_clean_nested_worktree.py::test_compile_clean_preserves_nested_git_worktree_agents_file	e2e

How to test

• Create a Git repository with an ignored nested linked worktree and commit an APM-marked AGENTS.md in that nested checkout.
• Create a marked stale AGENTS.md only in the parent checkout.
• Run apm compile --clean from the parent and confirm the parent orphan is removed.
• Compare nested file bytes and run git -C <nested> status --porcelain; both should remain unchanged and empty.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Copilot]

Pull request overview

This PR fixes apm compile --clean so it will not delete APM-generated AGENTS.md files that belong to nested Git worktrees under the project root, while still cleaning up stale/orphaned AGENTS.md files in the parent project. It also adds a real-CLI regression test, plus a static architecture boundary guard to prevent reintroducing an unprunable recursive scan.

Changes:

• Replace recursive Path.rglob("AGENTS.md") cleanup discovery with a prunable os.walk that stops descending into nested worktree roots (detected via .git file).
• Add an end-to-end lifecycle regression test that creates a nested worktree, commits an APM-marked AGENTS.md, runs compile --clean from the parent, and asserts nested bytes + clean git status.
• Add a lint architecture boundary check (and a mutation-style integration test) to ensure the worktree-pruning scan shape remains enforced; document the new --clean boundary behavior.

Show a summary per file

File	Description
src/apm_cli/compilation/distributed_compiler.py	Switch orphan discovery to os.walk and prune nested worktree roots before considering AGENTS.md for cleanup.
tests/integration/test_compile_clean_nested_worktree.py	Adds a real-CLI regression test covering nested-worktree preservation under compile --clean.
tests/integration/test_architecture_authorities.py	Adds a mutation-style guard test ensuring the architecture boundary check trips if the pruning predicate is removed.
scripts/lint-architecture-boundaries.sh	Adds an architecture boundary rule enforcing the prunable walk + .git-file boundary and rejecting reintroduction of rglob.
docs/src/content/docs/reference/cli/compile.md	Documents that --clean excludes nested Git worktrees from cleanup.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 2
• Review effort level: Lite

[danielmeppiel]

APM Review Panel: ship_with_followups

Nested worktree cleanup replaces rglob with os.walk pruning, backed by lifecycle e2e, an architecture mutation guard, and fast unit tests.

cc @sergio-sisternes-epam -- a fresh advisory pass is ready for your review.

Eight active panelists converged on a clean signal. All in-scope items were folded in 9098ad86d: explicit followlinks=False, one cleanup skip-dir constant, verbose boundary diagnostics, a fast .git-file unit proof, and the documented recovery command. The exact lifecycle test confirms that the parent orphan is removed while nested bytes remain identical and the nested Git status is empty.

One future placement-analysis concern is intentionally deferred: ContextOptimizer can still analyze non-hidden linked worktrees during normal placement generation. That is outside this PR's compile --clean orphan-deletion scope and predates this change.

Aligned with: secure by default through explicit non-following traversal; pragmatic package-manager behavior by honoring Git linked-worktree markers; OSS contributor workflows protected from silent cleanup loss.

Panel summary

Persona	B	R	N	Takeaway
Python Architect	0	1	1	Cleanup boundary is sound; normal placement analysis is a separate future surface.
CLI Logging Expert	0	0	0	No observable CLI output regression.
DevX UX Expert	0	1	2	Boundary diagnostics and recovery guidance are now present.
Supply Chain Security Expert	0	0	1	Cleanup containment improved; non-following walk is explicit.
OSS Growth Hacker	0	0	0	Fix builds trust for worktree-heavy contributors.
Doc Writer	0	0	0	CLI reference accurately states the cleanup boundary.
Test Coverage Expert	0	0	1	E2E, mutation, and unit coverage defend the user promise.
Performance Expert	0	0	2	Prunable traversal improves filesystem work without needing a benchmark.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Top 1 follow-up

1. [Python Architect] Audit ContextOptimizer traversal of non-hidden linked worktrees during placement generation -- it is a latent sibling surface outside this PR's cleanup scope.

Architecture

classDiagram
    direction LR
    class DistributedAgentsCompiler {
      <<Compiler>>
      +compile_distributed(primitives, config) CompilationResult
      -_find_orphaned_agents_files(generated, suppressed) list~Path~
      -_cleanup_orphaned_files(orphaned, dry_run) list~str~
      -_file_has_apm_marker(path) bool
    }
    class ContextOptimizer {
      <<Analyzer>>
      +analyze_directory_structure() None
      -_get_all_files() list~Path~
      -_should_exclude_subdir(path) bool
    }
    class CompilationResult {
      <<ValueObject>>
      +success bool
      +content_map dict
      +suppressed_empty_paths list~Path~
    }
    class DirectoryMap {
      <<ValueObject>>
      +directories dict
      +depth_map dict
    }
    DistributedAgentsCompiler *-- ContextOptimizer : delegates placement scoring
    DistributedAgentsCompiler ..> CompilationResult : returns
    ContextOptimizer ..> DirectoryMap : produces
    note for DistributedAgentsCompiler "PR changes _find_orphaned_agents_files: os.walk plus .git-file predicate replaces rglob"
    class DistributedAgentsCompiler:::touched
    classDef touched fill:#fff3b0,stroke:#d47600

Loading

flowchart TD
    A["apm compile --clean"] --> B["compile_distributed"]
    B --> C["_find_orphaned_agents_files"]
    C --> D["os.walk(self.base_dir)"]
    D --> E{"non-root .git file?"}
    E -- Yes --> F["clear child dirs and skip linked worktree"]
    E -- No --> G{"AGENTS.md present?"}
    G -- Yes --> H["marker-gated orphan candidate"]
    G -- No --> D
    H --> I["cleanup only when requested"]
    F --> D

Loading

Recommendation

Land after the current CI run is green. Track the ContextOptimizer placement-analysis audit separately; it does not change this cleanup fix's user promise.

---

Full per-persona findings

Python Architect

• [recommended] Audit ContextOptimizer traversal of non-hidden linked worktrees during normal placement generation. It is deferred because this PR changes compile --clean orphan deletion, not normal placement analysis.
• [nit] Confirmed the cleanup skip-dir set has one live owner after the fold.

CLI Logging Expert

No findings.

DevX UX Expert

• [recommended] Verbose nested-worktree boundary detail folded in 9098ad86d.
• [nit] Recovery guidance folded in 9098ad86d.
• [nit] The same recovery command is present in the --clean reference entry.

Supply Chain Security Expert

• [nit] Explicit followlinks=False folded in 9098ad86d.

OSS Growth Hacker

No findings.

Auth Expert -- inactive

No auth, token, credential, host classification, or authorization surface changed.

Doc Writer

No findings.

Test Coverage Expert

• [nit] Fast .git-file pruning unit proof folded in 9098ad86d; the exact lifecycle e2e and architecture mutation guard also pass.

Performance Expert

• [nit] Cleanup skip-dir constant folded in 9098ad86d.
• [nit] Explicit non-following walk folded in 9098ad86d.

_{This panel is advisory. It does not block merge. Re-apply the panel-review label after addressing feedback to re-run.}