Skip to content

feat: per-family drift notifications + stage-3 verification agent#457

Open
monsagri wants to merge 6 commits into
mainfrom
v3/stage3-verify-and-notifications
Open

feat: per-family drift notifications + stage-3 verification agent#457
monsagri wants to merge 6 commits into
mainfrom
v3/stage3-verify-and-notifications

Conversation

@monsagri

@monsagri monsagri commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

RUBBER STAMP - only edits to autogen pipeline that's not used in our main branch/release

Extends the API-coverage autogen pipeline (stages 1/2 shipped in #445/#446/#453) with two main-side workflow additions. Both are workflow-YAML only — no provider/tool code changes — and follow the established pattern of living on main (for workflow_dispatch registration) while operating on the preview-v3 line.

1. Per-family drift notifications (drift-report.yml)

Replaces the single truncated drift blob with one summary message plus one message per drifting family, posted individually to a generic {title, body} Slack Workflow Builder trigger:

  • Summary — drift counts (zero categories dropped) + scaffoldable families + a run link.
  • Per new family — its paths + a link/CLI to dispatch the stage-2 scaffold workflow for that family.
  • Per partial family with unclaimed operations — the lagging ops, flagged as needing a manual PR (stage 2 only scaffolds new resources).

No tool change: driftreport -format json already emits new_families / unclaimed_operations / status_counts and preserves exit-2-on-drift. Public-repo safety preserved — detail goes only to the private Slack channel (passed to curl, never echoed); the step summary stays yes/no.

2. Stage-3 verification agent (verify-scaffold.yml, new)

workflow_dispatch(pr_number) → verifies a stage-2 scaffolded draft PR against a real LaunchDarkly account before a human finishes it:

guard the PR (same-repo, scaffold/*, open) → check out the PR branch → fetch the LD token from SSM via AWS OIDC (same path as the acceptance matrix; api_host forced to prod) → build the provider to a pinned GOBIN with a terraform dev override → run claude-code-action to review the code and exercise the new resource with real terraform plan/apply, fixing functionality/tests until apply is clean. Applied example resources are retained (no destroy), namespaced tf-verify-pr<N> and pre-cleaned per run. Results post as a PR comment + a "ready for review" message on the same Slack trigger.

Security: prereq cleanup is a deterministic workflow step (the agent needs no curl with the live token); git tooling is scoped to the needed subcommands; runs are serialized per PR.

Validation

  • actionlint clean on both workflows; YAML parses.
  • Notification message-builder dry-run: summary + per-family messages render with working Slack mrkdwn links; verified live against the reshaped trigger.
  • Stage-3 build → dev-override → load → schema-validate → configure chain verified locally (stops at the auth boundary, as designed; CI supplies the SSM token).

Operator prerequisites

  • Done: the SLACK_DRIFT_WEBHOOK_URL Workflow Builder trigger reshaped to declare Text variables title + body (was message/repo/run_url/report); both flows + the stage-3 ping use it.
  • Before first verify run: set repository variable VERIFY_AGENT_ENABLED=true. The first run should be supervised (real applies under an admin token).

Notes / residual

  • GITHUB_TOKEN pushes don't re-trigger the PR's CI (re-run manually).
  • tf-verify-pr<N> resources are retained for review and not auto-swept (one set per PR; pre-deleted on re-run).
  • Weekly schedule: for the drift report and dropping the ref: preview-v3 checkout pins remain TODO for when v3 becomes main.

🤖 Generated with Claude Code


Note

High Risk
Stage 3 runs real terraform apply with a prod admin token and can push to PR branches; mis-guards or a bad dispatch could mutate live LD state or code outside intended scaffold PRs.

Overview
Drift reporting (stage 1) now emits JSON instead of markdown and replaces the single truncated Slack blob with multiple private-channel posts via a shared {title, body} Workflow Builder webhook: a category summary, one message per new API family (paths + scaffold dispatch link/CLI), one per scaffoldable net-new resource in a partial family, and informational posts for unclaimed ops. Public logs still only show yes/no drift.

Scaffold (stage 2) gains optional resource_name / operations inputs for scoped net-new resources, validation when scoped mode is incomplete, and agent prompts/branches/mapping instructions that follow whole-family vs scoped behavior.

New verify-scaffold.yml (stage 3) is a workflow_dispatch flow that guards same-repo open scaffold/* PRs, loads a prod LD token via AWS OIDC/SSM, builds the provider with a pinned GOBIN terraform dev override, pre-deletes tf-verify-pr<N> projects, runs a verification agent (real plan/apply, optional fixes pushed to the branch), and notifies Slack from verify-result.json.

Unrelated to autogen: generated vercel-native integration config adds an optional url field.

Reviewed by Cursor Bugbot for commit 5c81a79. Bugbot is set up for automated code reviews on this repo. Configure here.

monsagri and others added 2 commits June 15, 2026 17:37
Replace the single truncated drift-report blob with one summary message
plus one message per drifting family, posted individually to a generic
{title,body} Slack Workflow Builder trigger:

- summary: drift counts (zero categories dropped) + scaffoldable families
  + a run link;
- one per new endpoint family: paths + a link/CLI to dispatch the stage-2
  scaffold workflow for that family;
- one per partial family with unclaimed operations: the lagging ops, flagged
  as needing a manual PR (stage 2 only scaffolds new resources).

No tool change — driftreport already emits new_families / unclaimed_operations
/ status_counts in `-format json` and preserves exit-2-on-drift. Public-repo
safety preserved: family names / paths / operationIds go only to the private
Slack channel (passed to curl, never echoed); step summary stays yes/no.

Requires the SLACK_DRIFT_WEBHOOK_URL Workflow Builder trigger to declare two
Text variables `title` and `body` (was message/repo/run_url/report).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add verify-scaffold.yml: a workflow_dispatch(pr_number) job that verifies a
stage-2 scaffolded draft PR against a real LaunchDarkly account before a human
finishes it.

Flow: guard the PR (same-repo, scaffold/* branch, open, integer pr_number) →
check out the PR branch → fetch the LD token from SSM via AWS OIDC (the same
path the acceptance matrix uses; api_host forced to prod) → build the provider
to a pinned GOBIN with a terraform dev override → run claude-code-action to
review the code and exercise the new resource with real terraform plan/apply,
fixing functionality/tests until apply is clean. Applied example resources are
retained (no destroy), namespaced tf-verify-pr<N> and pre-cleaned per run.
Results post as a PR comment plus a "ready for review" message on the same
{title,body} Slack trigger as the drift report.

Gated by repository variable VERIFY_AGENT_ENABLED. Security: prereq cleanup is
a deterministic workflow step so the agent needs no curl with the live token;
git tooling is scoped to the needed subcommands; runs are serialized per PR.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire stage 1d (#458, merged to preview-v3) into the workflows on main.

drift-report.yml: the summary tally now counts scaffoldable_resources and
registered_candidates and lists candidate names alongside new families; a new
per-candidate message announces each curated net-new resource in a partial
family with a SCOPED stage-2 dispatch command (family + resource_name +
operations).

scaffold-resource.yml: new optional resource_name + operations dispatch inputs.
In scoped mode the agent implements exactly one net-new resource covering the
listed operations, branches/PRs under that name, and moves the ops out of the
family's new_resource_candidates into a resources entry (the family stays
partial) — explicitly NOT touching the family's existing resources. Whole-family
mode is unchanged. verify-scaffold.yml needs no change: a scaffold/<resource_name>
branch still matches its scaffold/* guard.

actionlint + YAML clean; the notification loop was dry-run against synthetic
scaffoldable_resources output.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread .github/workflows/verify-scaffold.yml
Comment thread .github/workflows/scaffold-resource.yml
monsagri and others added 2 commits June 16, 2026 11:13
Two bugbot findings on this PR:

- scaffold-resource.yml: SCOPED MODE (resource_name set) needs the operationId
  list, but nothing required it. Add a fail-fast guard step that errors when
  resource_name is set with empty operations, instead of letting the agent enter
  scoped mode with no ops and the whole-family slice.

- verify-scaffold.yml: the pre-clean step only deletes the tf-verify-pr<N>
  PROJECT, so an account-scoped resource reusing a stable name would 409 on a
  re-run — contradicting the "overwritten each run" comment. Account-scoped
  resources now get a per-run name (tf-verify-pr<N>-run<R>) so re-applies don't
  collide, and the header comment is corrected to state that project-scoped
  resources are pre-cleaned/overwritten while account-scoped ones accumulate and
  need manual cleanup.

actionlint + YAML clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 883a602. Configure here.

Comment thread .github/workflows/verify-scaffold.yml
Bugbot: the Resolve PR step wrote head_branch/pr_url/pr_title to GITHUB_OUTPUT
via plain `echo key=value`. A PR title (free text; a JSON string from `gh ...
--json title` can contain a newline) could inject a second `head_branch=…`
line. Actions takes the last duplicate key, so checkout could use a ref that
never passed the scaffold/* guard.

Strip CR/newlines from headRefName, url, and title right after extraction (so
the guard also sees the sanitized ref). head/url are single-line by
construction; title is the real vector. Verified: a title containing
`\nhead_branch=attacker-branch` is flattened to one line, leaving exactly one
head_branch output (the real branch).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant