Skip to content

fix(data): stop the permission repair following persisted symlinks - #165

Merged
Valtora merged 1 commit into
mainfrom
fix/permission-repair-symlink-follow
Jul 29, 2026
Merged

fix(data): stop the permission repair following persisted symlinks#165
Valtora merged 1 commit into
mainfrom
fix/permission-repair-symlink-follow

Conversation

@Valtora

@Valtora Valtora commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Description

repair_data_permissions walks the data directory and chmods every entry to 0700 or 0600. chmod dereferences, and os.walk yields a link-to-file under files and a link-to-directory under dirs, so any symlink stored under the data directory had its target re-permissioned instead, anywhere on the filesystem it pointed.

The Codex CLI keeps argv[0] dispatch links under CODEX_HOME/tmp/arg0 pointing back at its own binary inside the image, and Nojoin sets CODEX_HOME to a per-user directory on the persistent data volume because the subscription auth.json has to live there. Those links survive restarts, so a privileged pass set the bundled codex binary to 0600 and worker-io then failed every Codex-backed generation with EACCES.

The pass now skips symlinks: the directory list is pruned in place so the walk neither descends into a symlinked directory nor chmods it on the way past, and symlinked files are skipped before the chmod. Skipping is the only correction available, because Linux has no lchmod and os.chmod(..., follow_symlinks=False) raises NotImplementedError. Coreutils already behaves this way, so the chmod -R the deployment guide gives operators was never affected.

Under the gosu-dropped runtime user the chmod always failed with EPERM and only logged a warning, which is why the defect stayed latent in normal operation rather than being caught earlier. It becomes live the moment the pass runs privileged, which a root docker exec ... python ... does, since the worker image intentionally keeps USER root and backend.utils.config_manager builds its singleton at import.

Deliberately not paired with a cleanup of the Codex scratch directory, which was the other candidate fix. Codex reuses one lock-guarded arg0 directory rather than accumulating them, verified across repeated runs, and worker-io runs prefork with concurrency 4, so deleting that directory would race a live Codex process for no gain. TMPDIR does not relocate it either.

Existing deployments self-heal on upgrade: the 0600 lives in the container's writable overlay layer, so recreating worker-io restores 0755, and with this change it cannot be set again.

No new dependencies.

Fixes #164

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that changes existing behaviour)
  • Documentation update

Checks run

  • Backend tests: source .venv/bin/activate && pytest
  • Python quality: python scripts/check.py (Ruff lint, format check, mypy, doc and Alembic validators)
  • Frontend lint: cd frontend && npm run lint
  • Frontend unit tests: cd frontend && npm run test
  • Frontend build: cd frontend && npm run build
  • Docs validation: python3 scripts/validate_docs.py
  • Alembic validation: python3 scripts/validate_alembic.py

Migration impact

  • No database migration in this PR.
  • Adds an Alembic migration.

Documentation impact

  • No documentation change required.
  • Updated the relevant guide(s) in the same PR. The SEC-006 note in docs/DEPLOYMENT.md now states that the repair pass skips symbolic links.

Security impact

  • No security-sensitive change.
  • Touches auth, tokens, encryption, capture ownership, or exposure.

No auth, token, encryption or exposure boundary moves. The change does narrow a hardening gap as a side effect: while it stood, any symlink placed under the data directory gave the pass a chmod primitive over its target whenever the pass ran privileged. docs/SECURITY.md describes no behaviour that changes here, so it is untouched.

Manual verification

Reproduced and fixed inside the shipped nojoin-worker-io:local image, using the real Codex symlink target from the issue report and the real entrypoint sequence (root ownership repair, gosu drop to uid 1000).

Before, a root pass left the binary at uid=0 gid=0 mode=600 and codex failed to exec as uid 1000 with permission denied. After, the same root pass leaves it at uid=0 gid=0 mode=755 and codex-cli 0.145.0 runs as uid 1000.

Also confirmed against the real CLI rather than a synthetic fixture: Codex writes four links (apply_patch, applypatch, codex-execve-wrapper, codex-linux-sandbox), all pointing at the vendored binary named in the issue, and three consecutive runs produce one arg0 directory rather than three.

No capture, recording context-menu, or frontend surface touched.

repair_data_permissions walks the data directory and chmods every entry to
0700 or 0600. chmod dereferences, and os.walk yields a link-to-file under
files and a link-to-directory under dirs, so any symlink stored under the
data directory had its target re-permissioned instead, anywhere on the
filesystem it happened to point.

The Codex CLI puts argv[0] dispatch links under CODEX_HOME/tmp/arg0
pointing back at its own binary inside the image, and Nojoin sets
CODEX_HOME to a per-user directory on the persistent data volume because
the subscription auth.json has to live there. So the links persist across
restarts, and a privileged pass set the bundled codex binary to 0600.
worker-io then failed every Codex-backed generation with EACCES, which is
issue #164.

Reproduced in the shipped worker-io image and confirmed fixed there: a root
pass now leaves the binary at 0755 and it still runs as uid 1000. Under the
gosu-dropped runtime user the chmod always failed with EPERM and only
logged, which is why the defect stayed latent rather than being caught in
normal operation.

Skipping is the only correction available. Linux has no lchmod, so
os.chmod(..., follow_symlinks=False) raises NotImplementedError, and the
directory list has to be pruned in place so the walk neither descends into
a symlinked directory nor chmods it on the way past. Coreutils already
behaves this way, so the chmod -R the deployment guide gives operators was
never affected.

Not paired with a cleanup of the Codex scratch directory, which was the
other candidate fix. Codex reuses one lock-guarded arg0 directory rather
than accumulating them, verified across repeated runs, and worker-io runs
prefork with concurrency 4, so deleting that directory would race a live
Codex process for no gain. TMPDIR does not relocate it either.

Refs: #164, docs/DEPLOYMENT.md
@Valtora
Valtora merged commit 03da246 into main Jul 29, 2026
19 of 20 checks passed
@Valtora
Valtora deleted the fix/permission-repair-symlink-follow branch July 29, 2026 10:58
Copilot stopped work on behalf of Valtora due to an error July 29, 2026 11:14
Valtora added a commit that referenced this pull request Jul 30, 2026
Bump docs/VERSION to 2.2.0 so the tag validates, and fill in the
release-notes template for this range instead of hand-editing the
published body afterwards, which is how v2.1.0 was done.

A minor bump rather than a patch: #168 adds a capture action that did
not exist before, stopping and processing a paused recording.

Refs: #162, #163, #165, #167, #168, #169, #170, #171
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: Codex CLI OAuth inference fails with EACCES because permission repair follows persisted symlinks

1 participant