Skip to content

Commit 73251cb

Browse files
AbirAbbasclaude
AbirAbbas
and
claude
authored
feat(hitl): environment scout — negotiate scoped credentials before architecture (#78)
* feat(hitl): substrate for the environment scout (services + creds store + schema) Three new modules under swe_af/hitl/: - services.py — knowledge base of 9 common third-party services (Railway, Fly.io, Vercel, Supabase, Sentry, Datadog, GitHub, OpenAI, Anthropic) with their env var conventions, mint URLs, permissions hints, and signal files. Plus detect_services_from_repo() for a deterministic static pre-pass the LLM scout can build on. - credentials_store.py — process-local, execution-scoped dict for the credentials the scout negotiates. Keyed by run_id, thread-safe, isolates concurrent builds, NEVER persists. The full discussion of why this is in-memory (not BuildConfig, not app.memory, not the filesystem) lives in the module docstring. - scout_schema.py — ScoutResult Pydantic model used as the harness schema. Includes an explicit "scoped_credentials must NEVER round- trip through model_dump unless excluded" comment for callers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(hitl): run_environment_scout reasoner + wire into plan() and harness env Adds the new reasoner that runs once between PM and Architect when HAX is enabled. The scout reads the PRD + repo, identifies third-party services whose absence would block the work, and asks the user for scoped / temporary tokens via a single Hax mega-form. Submitted values are stashed in the in-memory credentials store keyed by run_id; the scout's return payload OMITS scoped_credentials so the secrets never reach the control- plane workflow_execution row. - swe_af/prompts/environment_scout.py — system prompt + task-prompt builder. Strong guidance on when NOT to ask (purely local PRD, prior answers already cover the question, no genuine PRD-blocking requirement). - swe_af/reasoners/pipeline.py — @router.reasoner async def run_environment_scout. Same wrapper shape as the three reasoners from PR #77; uses run_with_ask_user with budget=2. - swe_af/app.py: * plan() — Phase 1.5 calls run_environment_scout via app.call BETWEEN PM and architect; guarded so it runs only when HAX_API_KEY is set. * build() body wrapped in try/finally so clear_scoped_credentials ALWAYS runs on exit (success or exception). Eliminates secret leakage across builds within the same agent process. * app.harness is monkey-patched once at module load to auto-inject stored credentials as env vars on EVERY harness call across the pipeline. Avoids touching the 25+ existing call sites. Backwards-compatible: with HAX_API_KEY unset, plan() skips the scout and the monkey-patched harness passes os.environ through unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(hitl): 17 unit tests for the environment-scout substrate Three pillars covered: - services.py — KNOWN_SERVICES inventory bounds, missing-path safety, file + directory signal detection, prompt-summary rendering. - credentials_store.py — round-trip, blank/None filtering, isolation between execution_ids, get-returns-copy, concurrent thread safety, inject-into-env layering rules. - scout closure round-trip — pass 1 emits ask_user_form via the wrapper, pass 2 sees prior_user_responses and returns scoped_credentials; no-services-detected short-circuits the pause; model_dump(exclude={"scoped_credentials"}) actually strips the field. All tests mock HaxClient + app.pause; no real network, no real harness. Pin a baseline of 8+ services so future trimming is visible in diff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 0a4c3b7 commit 73251cb

8 files changed

Lines changed: 1704 additions & 722 deletions

File tree

‎swe_af/app.py‎

Lines changed: 786 additions & 722 deletions
Large diffs are not rendered by default.

‎swe_af/hitl/__init__.py‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,17 +10,39 @@
1010
format_prior_user_responses,
1111
request_user_input_and_pause,
1212
)
13+
from swe_af.hitl.credentials_store import (
14+
clear_scoped_credentials,
15+
get_scoped_credentials,
16+
inject_credentials_into_env,
17+
store_scoped_credentials,
18+
)
19+
from swe_af.hitl.scout_schema import ScoutResult
20+
from swe_af.hitl.services import (
21+
KNOWN_SERVICES,
22+
ServiceCredentialSpec,
23+
detect_services_from_repo,
24+
known_service_summary_for_prompt,
25+
)
1326
from swe_af.hitl.wrapper import AskUserBudget, run_with_ask_user
1427

1528
__all__ = [
1629
"AskUserForm",
1730
"AskUserFormField",
1831
"AskUserResponse",
1932
"AskUserBudget",
33+
"KNOWN_SERVICES",
34+
"ScoutResult",
35+
"ServiceCredentialSpec",
2036
"approval_webhook_url",
2137
"build_form_builder",
2238
"build_hax_client_from_env",
39+
"clear_scoped_credentials",
40+
"detect_services_from_repo",
2341
"format_prior_user_responses",
42+
"get_scoped_credentials",
43+
"inject_credentials_into_env",
44+
"known_service_summary_for_prompt",
2445
"request_user_input_and_pause",
2546
"run_with_ask_user",
47+
"store_scoped_credentials",
2648
]

‎swe_af/hitl/credentials_store.py‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
"""Process-local, execution-scoped store for credentials the scout negotiates.
2+
3+
Why a module-level dict instead of ``BuildConfig`` or ``app.memory``:
4+
5+
* ``BuildConfig`` is serialized through ``to_execution_config_dict()`` and
6+
passed to ``execute()`` via ``app.call``. The control plane logs all
7+
``app.call`` input data, which would persist the credentials.
8+
* ``app.memory`` (scope=``run``) is synced to the control plane DB by design
9+
— also persists.
10+
* Filesystem under ``artifacts_dir`` is written to disk and archived.
11+
12+
The scout's negotiation produces credentials that should *only* live in the
13+
agent process's memory for the duration of the build, then be cleared. A
14+
module-level dict keyed by execution_id is the simplest way to achieve that
15+
while keeping concurrent builds (which share the Python process) isolated.
16+
17+
Security boundary:
18+
19+
* Values are never logged.
20+
* Values are never written to disk.
21+
* Values are not serialized through ``app.call`` (use this store from inside
22+
the receiving reasoner, not as a kwarg).
23+
* The build()'s ``finally`` block MUST call ``clear_scoped_credentials`` —
24+
every error path included.
25+
"""
26+
27+
from __future__ import annotations
28+
29+
import threading
30+
31+
# Module-level. Keyed by execution_id (each build has its own).
32+
_STORE: dict[str, dict[str, str]] = {}
33+
_LOCK = threading.Lock()
34+
35+
36+
def store_scoped_credentials(execution_id: str, creds: dict[str, str]) -> None:
37+
"""Replace the stored credentials for ``execution_id`` with ``creds``.
38+
39+
Filters out None/empty values so a partially-filled mega-form (user skipped
40+
some fields) doesn't surface as empty env vars to downstream subprocesses
41+
(which can be confusing — "is the env set or not?").
42+
"""
43+
if not execution_id:
44+
return
45+
filtered = {
46+
k: v
47+
for k, v in (creds or {}).items()
48+
if isinstance(v, str) and v.strip()
49+
}
50+
with _LOCK:
51+
if filtered:
52+
_STORE[execution_id] = filtered
53+
else:
54+
_STORE.pop(execution_id, None)
55+
56+
57+
def get_scoped_credentials(execution_id: str) -> dict[str, str]:
58+
"""Return a *copy* of the stored credentials for ``execution_id``.
59+
60+
Returns an empty dict if nothing is stored — callers should treat that as
61+
"no credentials negotiated; rely on os.environ only".
62+
"""
63+
if not execution_id:
64+
return {}
65+
with _LOCK:
66+
stored = _STORE.get(execution_id)
67+
return dict(stored) if stored else {}
68+
69+
70+
def clear_scoped_credentials(execution_id: str) -> None:
71+
"""Remove credentials for ``execution_id`` from process memory."""
72+
if not execution_id:
73+
return
74+
with _LOCK:
75+
_STORE.pop(execution_id, None)
76+
77+
78+
def inject_credentials_into_env(
79+
base_env: dict[str, str] | None, execution_id: str
80+
) -> dict[str, str]:
81+
"""Return a NEW env dict = ``base_env`` ∪ scoped credentials.
82+
83+
Scoped credentials WIN over ``base_env`` so a freshly-minted token from
84+
the scout overrides any stale value already in os.environ (e.g. an
85+
expired RAILWAY_TOKEN from a previous build).
86+
87+
Callers should use this immediately before each ``router.harness(...)``
88+
call, passing the result as the ``env=`` kwarg. The base is normally
89+
``dict(os.environ)`` so the subprocess still inherits everything the
90+
parent has — we only ADD/override the scoped creds.
91+
"""
92+
merged: dict[str, str] = dict(base_env or {})
93+
creds = get_scoped_credentials(execution_id)
94+
if creds:
95+
merged.update(creds)
96+
return merged

‎swe_af/hitl/scout_schema.py‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
"""Structured output schema for ``run_environment_scout``.
2+
3+
The scout is a two-pass reasoner driven by ``run_with_ask_user``:
4+
5+
* Pass 1 (no ``prior_user_responses``): scan the repo, populate
6+
``detected_services`` and ``ask_user_form`` with one optional text field
7+
per service. ``scoped_credentials`` stays empty.
8+
* Pass 2 (after the user submits): take the values from
9+
``prior_user_responses[-1]['values']`` and surface them as
10+
``scoped_credentials``. ``ask_user_form`` is cleared.
11+
12+
If no services are detected on pass 1, the scout returns
13+
``ask_user_form=None`` immediately and the wrapper short-circuits — no pause,
14+
no second pass.
15+
"""
16+
17+
from __future__ import annotations
18+
19+
from pydantic import BaseModel, Field
20+
21+
from swe_af.hitl.ask_user import AskUserForm
22+
from swe_af.hitl.services import ServiceCredentialSpec
23+
24+
25+
class ScoutResult(BaseModel):
26+
"""Structured output the scout LLM emits."""
27+
28+
detected_services: list[ServiceCredentialSpec] = Field(
29+
default_factory=list,
30+
description=(
31+
"Third-party services the scout believes the PRD work touches. "
32+
"On pass 1 this matches the form's fields one-for-one."
33+
),
34+
)
35+
scoped_credentials: dict[str, str] = Field(
36+
default_factory=dict,
37+
description=(
38+
"Populated on pass 2 ONLY. Keys are env var names (matching "
39+
"ServiceCredentialSpec.env_var_name); values are the secrets the "
40+
"user provided. Must NOT be logged or persisted."
41+
),
42+
)
43+
skipped_services: list[str] = Field(
44+
default_factory=list,
45+
description=(
46+
"Env var names the user explicitly left blank (informed opt-out). "
47+
"Surfaced so downstream code can warn early if a critical "
48+
"credential is missing."
49+
),
50+
)
51+
summary: str = Field(
52+
default="",
53+
description=(
54+
"One-line summary the scout writes — e.g. 'Negotiated 2 "
55+
"credentials: RAILWAY_TOKEN, SENTRY_AUTH_TOKEN. User skipped: "
56+
"DATADOG_API_KEY.' Safe to log; never includes secret values."
57+
),
58+
)
59+
ask_user_form: AskUserForm | None = None

‎swe_af/hitl/services.py‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
"""Knowledge base of common third-party services + how to mint a scoped token.
2+
3+
Used by ``run_environment_scout`` to recognize signal files in a repo (e.g.
4+
``railway.toml``, ``fly.toml``, ``sentry.properties``) and ask the user for
5+
the matching scoped credential. The LLM inside the scout reasoner consumes
6+
``KNOWN_SERVICES`` as a hint list; ``detect_services_from_repo`` provides a
7+
deterministic pre-pass that the LLM can build on.
8+
"""
9+
10+
from __future__ import annotations
11+
12+
import os
13+
from typing import Iterable
14+
15+
from pydantic import BaseModel, Field
16+
17+
18+
class ServiceCredentialSpec(BaseModel):
19+
"""One row in the knowledge base; also returned by the scout."""
20+
21+
service_name: str = Field(description="Human-readable service name shown to the user.")
22+
env_var_name: str = Field(
23+
description=(
24+
"Env var the build expects (becomes the ask_user_form field id). "
25+
"Match what the service's CLI / SDK looks for by default."
26+
)
27+
)
28+
mint_url: str = Field(
29+
description=(
30+
"URL where the user mints a scoped/temporary token. Surfaced in "
31+
"the form description so the user can click through and paste back."
32+
)
33+
)
34+
permissions_hint: str = Field(
35+
description=(
36+
"Short hint on what scope / TTL to request when minting. Shown to "
37+
"the user in the form so they don't over-grant access."
38+
)
39+
)
40+
signal_files: list[str] = Field(
41+
default_factory=list,
42+
description=(
43+
"Glob-ish filenames whose presence in the repo strongly implies "
44+
"this service is in use. Used by detect_services_from_repo."
45+
),
46+
)
47+
evidence_template: str = Field(
48+
default="",
49+
description=(
50+
"Sentence template explaining WHY the build needs this credential, "
51+
"used in the form description. Use {{signal}} as the placeholder."
52+
),
53+
)
54+
55+
56+
KNOWN_SERVICES: list[ServiceCredentialSpec] = [
57+
ServiceCredentialSpec(
58+
service_name="Railway",
59+
env_var_name="RAILWAY_TOKEN",
60+
mint_url="https://railway.com/account/tokens",
61+
permissions_hint="Project token, read-only if possible, set expiry to 1 day.",
62+
signal_files=["railway.toml", "railway.json", ".railway/config.json"],
63+
evidence_template="Saw {signal} — build likely needs Railway access to deploy or query services.",
64+
),
65+
ServiceCredentialSpec(
66+
service_name="Fly.io",
67+
env_var_name="FLY_API_TOKEN",
68+
mint_url="https://fly.io/user/personal_access_tokens",
69+
permissions_hint="Deploy token scoped to this app, 1-day expiry.",
70+
signal_files=["fly.toml", "fly.io.toml", ".fly/config.toml"],
71+
evidence_template="Saw {signal} — build may need Fly.io access for deploys.",
72+
),
73+
ServiceCredentialSpec(
74+
service_name="Vercel",
75+
env_var_name="VERCEL_TOKEN",
76+
mint_url="https://vercel.com/account/tokens",
77+
permissions_hint="Scope to this team only, 1-day expiry.",
78+
signal_files=["vercel.json", ".vercel/project.json"],
79+
evidence_template="Saw {signal} — build may need Vercel access.",
80+
),
81+
ServiceCredentialSpec(
82+
service_name="Supabase",
83+
env_var_name="SUPABASE_ACCESS_TOKEN",
84+
mint_url="https://supabase.com/dashboard/account/tokens",
85+
permissions_hint="Personal access token, 1-day expiry — required only if migrations or schema changes are part of the work.",
86+
signal_files=["supabase/config.toml", "supabase/.gitignore", "supabase/migrations"],
87+
evidence_template="Saw {signal} — Supabase project detected.",
88+
),
89+
ServiceCredentialSpec(
90+
service_name="Sentry",
91+
env_var_name="SENTRY_AUTH_TOKEN",
92+
mint_url="https://sentry.io/settings/account/api/auth-tokens/",
93+
permissions_hint="Auth token scoped to project:read + project:releases, 1-day expiry.",
94+
signal_files=["sentry.properties", ".sentryclirc", "sentry.io.json"],
95+
evidence_template="Saw {signal} — Sentry integration detected.",
96+
),
97+
ServiceCredentialSpec(
98+
service_name="Datadog",
99+
env_var_name="DATADOG_API_KEY",
100+
mint_url="https://app.datadoghq.com/organization-settings/api-keys",
101+
permissions_hint="Application API key (NOT a client token), restricted to read scopes if possible.",
102+
signal_files=["datadog.yaml", ".datadog/conf.yaml"],
103+
evidence_template="Saw {signal} — Datadog integration detected.",
104+
),
105+
ServiceCredentialSpec(
106+
service_name="GitHub",
107+
env_var_name="GH_TOKEN",
108+
mint_url="https://github.com/settings/personal-access-tokens/new",
109+
permissions_hint="Fine-grained PAT scoped to THIS repo only, repo:contents+pull-requests, 1-day expiry.",
110+
signal_files=[".github/workflows", "CODEOWNERS"],
111+
evidence_template="Saw {signal} — work likely needs GitHub API beyond what gh CLI provides anonymously.",
112+
),
113+
ServiceCredentialSpec(
114+
service_name="OpenAI",
115+
env_var_name="OPENAI_API_KEY",
116+
mint_url="https://platform.openai.com/api-keys",
117+
permissions_hint="Restricted API key with low usage cap; 1-day expiry.",
118+
signal_files=[], # Detected via dependency manifests, not signal files.
119+
evidence_template="Project depends on the OpenAI SDK.",
120+
),
121+
ServiceCredentialSpec(
122+
service_name="Anthropic",
123+
env_var_name="ANTHROPIC_API_KEY",
124+
mint_url="https://console.anthropic.com/settings/keys",
125+
permissions_hint="Restricted API key, set monthly spend limit, 1-day expiry.",
126+
signal_files=[],
127+
evidence_template="Project depends on the Anthropic SDK.",
128+
),
129+
]
130+
131+
132+
def detect_services_from_repo(repo_path: str) -> list[ServiceCredentialSpec]:
133+
"""Deterministic pre-pass: look for ``signal_files`` under ``repo_path``.
134+
135+
Returns the subset of ``KNOWN_SERVICES`` whose signal files exist on disk.
136+
This is a hint to the LLM scout — the final decision on which credentials
137+
to ask for stays with the scout, which can incorporate PRD context the
138+
static scan can't see.
139+
140+
Notes:
141+
* No recursive glob; checks each ``signal_file`` as a path under
142+
``repo_path``. ``signal_file`` may be a file or a directory; both
143+
count as a hit.
144+
* Returns an empty list if ``repo_path`` doesn't exist (don't raise).
145+
* Order matches ``KNOWN_SERVICES`` so callers get stable output.
146+
"""
147+
if not repo_path or not os.path.isdir(repo_path):
148+
return []
149+
hits: list[ServiceCredentialSpec] = []
150+
for spec in KNOWN_SERVICES:
151+
for signal in spec.signal_files:
152+
candidate = os.path.join(repo_path, signal)
153+
if os.path.exists(candidate):
154+
hits.append(spec)
155+
break
156+
return hits
157+
158+
159+
def known_service_summary_for_prompt(specs: Iterable[ServiceCredentialSpec]) -> str:
160+
"""Render a markdown bullet list of service specs for inclusion in a prompt."""
161+
lines: list[str] = []
162+
for spec in specs:
163+
signals = ", ".join(f"`{s}`" for s in spec.signal_files) or "(no static signal)"
164+
lines.append(
165+
f"- **{spec.service_name}** — env `{spec.env_var_name}`; "
166+
f"signals: {signals}; mint at {spec.mint_url}; "
167+
f"hint: {spec.permissions_hint}"
168+
)
169+
return "\n".join(lines)

0 commit comments

Comments
 (0)