[audit-workflows] Daily Agentic Workflow Audit — 2026-08-05 (94% success, memory gap closed)

[github-actions[bot]]

Overview

24h fleet audit (2026-08-05, window -1d): 100 runs, 94 success, 6 failure — 94.0% raw success rate. No intentional-failure workflows (Daily Credit Limit Test, Daily Max AI Credits Test) ran in this window, so the adjusted rate is also 94.0%. All 6 failures are driver-exit failures (harness/infra-level) — zero agent-logic failures, zero missing-tools, zero missing-data.

Engine mix: copilot 49, pi 25, claude 17, codex 7, crush 1, goose 1.

A significant secondary finding: this audit's own repo-memory had gone stale for ~30 days (last write 2026-07-06) even though the underlying audit-workflows GitHub Actions workflow kept running daily and mostly succeeding — see Known-issue: repo-memory gap below. This cycle backfills that memory and closes the gap.

Key metrics

Metric	Value
Total runs	100
Success / Failure	94 / 6
Success rate (raw / adjusted)	94.0% / 94.0%
Total tokens	6,088,275
Action minutes	1,027
GitHub API calls	2,181
Driver-exit failures	6
Agent-logic failures	0

Failure clusters (4 distinct root causes)

1. Copilot subagent "No model available" (recurrence of known issue, now seen 25×) — run 31032399483, run 31030978505 (PR Code Quality Reviewer). The Copilot task-tool subagent fails all retries with Execution failed: Error: No model available. Check policy enablement under GitHub Settings > Copilot, exhausting 3 retries then exit code 1.
2. NEW — Claude Code max-invocations cap hit (429): Design Decision Gate run 31039435144 hit Maximum LLM invocations exceeded (20/20) after 10 consecutive 429 rate-limit retries — the transcript shows the agent burning turns trying ~7 different shell syntaxes to call safeoutputs add_comment before running out of budget. Avenger run 31037177175 hit Maximum LLM invocations exceeded (25/25) with no further retry attempted despite retriesRemaining=3.
3. NEW — Goose harness config bug: Daily Team Evolution Insights run 31044930068 crashes immediately with Error: GH_AW_ENGINE_VERSION is required at goose_harness.cjs:12 — 0 turns, 0 tokens, before the agent ever starts. Likely a stale .lock.yml.
4. NEW — evals.jsonl push race: Auto-Triage Issues run 31030564682 — the agent itself succeeded; only the post-run push_evals_state step failed after 4 retries on a git rebase conflict from a concurrent writer to evals.jsonl.

Full per-run failure detail

• 31032399483 / 31030978505 (PR Code Quality Reviewer, copilot): step "Execute GitHub Copilot CLI" fails; agent-stdio.log shows [copilot-sdk-driver] [sdk-driver] error: Execution failed: Error: No model available. Check policy enablement under GitHub Settings > Copilot repeated on every retry; harness gives up after exhausting retries (exitCode=1).
• 31039435144 (Design Decision Gate, claude): 10× api_retry events (error_status:429, error:"rate_limit") then terminal API Error: Request rejected (429) · Maximum LLM invocations exceeded (20 / 20). permission_denials array shows repeated attempts to post a comment via safeoutputs add_comment using jq -Rs, python3 -c, printf, direct cat piping — none landing before the cap was hit.
• 31037177175 (Avenger, claude): API Error: Request rejected (429) · Maximum LLM invocations exceeded (25 / 25). [claude-harness] log: isRateLimitError=true, permissionDeniedCount=6, hasNumerousPermissionDenied=true, retriesRemaining=3 — but the run ends immediately (done: exitCode=1) with no further retry attempted.
• 31044930068 (Daily Team Evolution Insights, goose): agent-stdio.log shows a pre-agent crash: Error: GH_AW_ENGINE_VERSION is required at /home/runner/work/_temp/gh-aw/actions/goose_harness.cjs:12:21. No agent turns executed.
• 31030564682 (Auto-Triage Issues, pi): run_summary.json shows the agent step succeeded; the failing step is "Push evals results to git" (push_evals_state). Logs show 3× Push failed (attempt N/4), retrying...: pushSignedCommits: failed to rebase commit range onto current GraphQL parent... then Failed to push evals results after 4 attempts... Root cause: Auto-merging evals.jsonl.

Known-issue: repo-memory gap (audit self-check)

This audit's repo-memory (audit-history.jsonl, metrics-summary.json, known-issues.json, etc., under memory/audit-workflows) was last updated 2026-07-06. A fresh query scoped to workflow_name: audit-workflows, start_date: -30d shows the workflow kept running daily through 2026-08-04 (~30 runs, mostly successful) — meaning this audit ran but silently stopped persisting findings for roughly a month. Every recurrence/persistence counter carried forward from before the gap is therefore an undercount. This cycle's memory update backfills known-issues, recommendations, anomalies, workflow-trends, metrics-summary, and audit-history, and flags the gap itself as a new HIGH-severity issue (repo-memory-audit-persistence-gap) so it gets tracked going forward.

Trend chart 1 — Workflow Health (success/failure counts + success rate)

The shaded band marks the ~30-day repo-memory gap (2026-07-06 → 2026-08-05) called out above; the line is deliberately broken across it rather than interpolated, since no daily data exists for that stretch. The last two real data points — 2026-07-06 (86.25% over 400 runs) and today (94.0% over 100 runs) — both sit comfortably in the "healthy" band with all failures being driver-exit rather than agent-logic issues, so the fleet's underlying reliability looks stable across the gap even though it wasn't being measured.

Trend chart 2 — Token Usage (daily + moving average)

Same gap treatment applies here. Today's 6.09M tokens over 100 runs (~61K tokens/run average) is in line with the pre-gap daily volumes (3.3–3.4M tokens over ~100-400 runs on 2026-07-05/06), so there's no evidence of a token-usage regression — the gap is a reporting blind spot, not a cost anomaly.

Next actions

1. Investigate the repo-memory persistence gap (repo-memory-audit-persistence-gap) — check push_repo_memory step logs on a run from the gap window (e.g. 2026-08-04) to see whether writes were attempted and rejected, or never attempted at all.
2. Escalate the Copilot "No model available" policy gap (copilot-sdk-driver-failures, now 25 recurrences) to whoever owns Copilot policy enablement — this is the fleet's single most persistent failure class.
3. Patch the Goose .lock.yml for Daily Team Evolution Insights to supply GH_AW_ENGINE_VERSION — quick fix, currently 100%-failing before the agent even starts.
4. Decide on a fix for Claude Code max-invocations exhaustion — either raise the cap for comment/report-writing workflows or have the harness flush partial safe-output before hard-failing at the cap; also worth clarifying the safeoutputs add_comment CLI invocation so agents don't burn turns guessing syntax.
5. Add retry/rebase-with-backoff to push_evals_state to avoid the evals.jsonl concurrent-write race (low priority — telemetry-only impact).

References:

• Run 31032399483 — PR Code Quality Reviewer
• Run 31039435144 — Design Decision Gate
• Run 31044930068 — Daily Team Evolution Insights

Generated by 🔍 Agentic Workflow Audit Agent · agent · 324.3 AIC · ⌖ 47.4 AIC · ⊞ 8.4K · ◷

• expires on Aug 6, 2026, 1:58 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-06T21:58:12.513Z.

Closed by Workflow