[prompt-clustering] Prompt Clustering Analysis - 2026-08-06

[github-actions[bot]]

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Summary

Analysis Period: Last 30 days (2026-07-20 → 2026-08-06)
Total Tasks Analyzed: 1,000 copilot-agent PRs (all usable — 748 merged, 248 closed, 4 open)
Clusters Identified: 6
Overall Success Rate: 74.8%

Full Analysis Report

Methodology

• Pulled full PR data (title, body, comments, reviews, commits, file stats) for all 1,000 copilot PRs from the last 30 days.
• Cleaned PR bodies: stripped code fences and URLs, and — important fix — cut text at the <!-- START COPILOT CODING AGENT SUFFIX --> marker, since that HTML comment delimits the end of the human-authored task description, not the start (an earlier draft of this pipeline had this backwards and discarded ~46% of prompts as "too short"; corrected before this run).
• Vectorized title+body with TF-IDF (unigrams+bigrams, 150 features, min_df=3, max_df=0.6).
• Ran K-means for k=4..8 and picked k=6 by silhouette score (0.058, the best balance of cohesion and interpretable, non-trivial cluster sizes — scores rise slowly through k=8 but split clusters that are still thematically identical).
• Limitation: workflow run logs (gh-aw logs) were not ingested for turn-count/cost metrics — downloading logs for ~1,000 runs across 30 days was not practical within this run's time budget. This report is based on PR text and interaction data (comments, reviews, files changed) only, not agent turn counts or token cost.

General Insights

• Most common task type: Cluster 5, "Agentic Workflow Infra" (379 tasks, 37.9% of all PRs) — changes to workflow YAML/markdown, agentic tooling, and copilot/gh-aw runtime behavior.
• Highest success rate: Cluster 2, "Footer/Metadata (Sous Chef)" at 90.0% — small, mechanical, well-scoped changes (footer rendering, container-image substitutions, static-analysis additions).
• Lowest success rate: Cluster 3, "Container, MCP & Publishing" at 56.4% — infra/publishing changes (container pins, multi-platform image publishing, security scans) that also carry the second-highest average files-changed (35.2) and highest closed-without-merge count relative to size (34 of 78).
• Most reviewed per-task: Cluster 2 again — 12.2 avg comments and 5.85 avg reviews per PR, well above the ~6.0 comment / ~3.0 review dataset average, despite its high merge rate. These are heavily scrutinized despite being high-success, suggesting reviewers still double-check "footer/metadata"-labeled changes carefully even when they're usually correct.
• Largest average diff size: Cluster 2 also leads on avg_files_changed (58.75) — somewhat surprising for a cluster whose top terms suggest small mechanical edits; likely driven by a handful of broader "sous chef" refactors bundled with metadata work.

Cluster Analysis

Cluster 1 (displayed as "Agentic Workflow Infra"): Workflow & Agentic Runtime Changes

• Size: 379 tasks (37.9% of total)
• Success Rate: 72.8%
• Avg Comments / Reviews: 4.2 / 2.2
• Top Keywords: workflow, md, workflows, github, agent, copilot, aw, run
• Characteristics: The largest and most generic cluster — changes to .github/workflows/*, markdown-based workflow definitions, and gh-aw/copilot runtime behavior (timeouts, gateway config, driver switches).
• Example PRs: fix(cli): address 16 CLI consistency issues from 2026-07-20 audit #46854, fix: propagate tools.startup-timeout to gateway.startupTimeout (prevents safeoutputs eviction on slow runners) #46875, docs(spdd): tighten normative language, add safeguards, expand glossary (rotation 34–38) #46886, daily-issues-report: switch Copilot SDK driver from Python to Node in workflow runtime #46894, [WIP] Fix normalization of report formatting for non-compliant workflows #46896

---

Cluster 2 (displayed as "Bug Fixes, Tests & CI"): Bug Fixes, Tests & CI Failures

• Size: 293 tasks (29.3% of total)
• Success Rate: 80.2%
• Avg Comments / Reviews: 7.5 / 3.8
• Top Keywords: cjs, test, job, error, tests, fix, added, actions
• Characteristics: Classic "fix a bug / fix a failing CI job / add a test" tasks. Second-highest success rate among non-trivial clusters, and highest review intensity of the four large clusters — consistent with bug-fix PRs needing more back-and-forth to confirm root cause.
• Example PRs: fix: nest issue-intent metadata correctly in close_issue and assign_to_agent payloads #46850, Recognize global.getOctokit() aliases in no-github-request-interpolated-route #46856, Clarify intent-aware safe-output mutation contracts across optional and strict modes #46857, [WIP] Fix failing GitHub Actions job for CLI completion #46859, feat: mode-specific intent guidance for all safe-output mutation tools #46861

---

Cluster 3 (displayed as "CLI & Package Features"): CLI & Package-Level Features

• Size: 122 tasks (12.2% of total)
• Success Rate: 75.4%
• Avg Comments / Reviews: 6.9 / 3.3
• Top Keywords: pkg, cli, pkg cli, pkg workflow, workflow, changes, new, file
• Characteristics: Go-package-level CLI feature work and refactors (flag behavior, linter registries, parser extraction). Smallest average diff size (13.7 files) of the six clusters — targeted, surgical changes.
• Example PRs: feat(init): make --codespaces optional-value; grant discussions/issues write #46849, Refactor bootstrap profile runner into focused modules #46855, Rename CLI firewall PolicyRule to FirewallPolicyRule #46961, fix: expose linters.All() registry, document sprintfbool, add bidirectional doc-sync guard #47007, refactor(parser): extract extractEngineMCPSettings to bring extractEngineConfig under 60-line limit #47009

---

Cluster 4 (displayed as "Container, MCP & Publishing"): Container, MCP & Publishing Infra

• Size: 78 tasks (7.8% of total)
• Success Rate: 56.4% — lowest of all clusters
• Avg Comments / Reviews: 4.9 / 1.9
• Top Keywords: image, container, aw, mcp, gh, gh aw, scan, json
• Characteristics: Container image pinning, multi-platform publishing, MCP config, and security-scan integration. Large average diffs (35.2 files) combined with the lowest success rate suggests this category of task is either under-specified in the originating prompts or touches infra that's harder for an agent to validate locally (registries, multi-arch builds, scan tooling) before a human can merge.
• Example PRs: chore: CLI version updates 2026-07-21 #47005, feat: add container_pins to aw.json and document action/container substitutions #47351, Refresh gh-aw-node packages and multi-platform publishing #47405, Add daily firewall container security scan #47407, Fix container scan exit handling and failure propagation #47420

---

Cluster 5 (displayed as "AI Engine & Docs"): AI Engine Config & Docs

• Size: 68 tasks (6.8% of total)
• Success Rate: 69.1%
• Avg Comments / Reviews: 3.6 / 1.9
• Top Keywords: ai, agent, txt, docs, engine, agentic, md, daily
• Characteristics: Engine-switching (codex↔copilot), AI-credit budget/pricing config, and docs/SEO work (llms.txt, robots.txt). Lowest average review count — likely lower-risk, docs-adjacent changes that don't need as much scrutiny.
• Example PRs: [WIP] Add robots.txt and llms.txt to docs site #46893, Switch AI Moderator engine from codex to copilot #46928, Add Copilot AI-credits fallback pricing and skip AI-credits guardrails for BYOK #46986, docs: prerender /llms.txt for GitHub Pages AI indexing #47121, fix(daily-vulnhunter-scan): raise max-ai-credits to 1500 #47236

---

Cluster 6 (displayed as "Footer/Metadata (Sous Chef)"): Footer/Metadata & Static Analysis

• Size: 60 tasks (6.0% of total)
• Success Rate: 90.0% — highest of all clusters
• Avg Comments / Reviews: 12.2 / 5.9 — highest of all clusters
• Top Keywords: sous, sous chef, chef, pr sous, pr, em, sub, aic
• Characteristics: Footer-rendering metadata (AIC display), container-image substitution rules, and static-analysis tooling additions (yamllint, syft scans). High scrutiny, high success — the review process here appears to be working well.
• Example PRs: Render compressed model alias before AIC in generated footers #47018, Mitigate safe-outputs review failures when inline comment anchors are partially invalid #47356, Add container image substitutions for private registries #47374, Add yamllint as a static analysis validator #47507, Add --syft container scan support to gh aw compile #47515

Success Rate by Cluster

Cluster	Tasks	Success Rate	Avg Comments	Avg Reviews	Avg Files Changed	Top Keywords
Agentic Workflow Infra	379	72.8%	4.2	2.2	22.3	workflow, agent, copilot, aw
Bug Fixes, Tests & CI	293	80.2%	7.5	3.8	20.0	test, job, error, fix
CLI & Package Features	122	75.4%	6.9	3.3	13.7	pkg, cli, workflow, file
Container, MCP & Publishing	78	56.4%	4.9	1.9	35.2	image, container, mcp, scan
AI Engine & Docs	68	69.1%	3.6	1.9	21.5	ai, agent, docs, engine
Footer/Metadata (Sous Chef)	60	90.0%	12.2	5.9	58.8	sous chef, aic, pr

Full Data Table (sample of 20 PRs across clusters)

PR #	Title	Cluster	Outcome
46854	fix(cli): address 16 CLI consistency issues from 2026-07-20 audit	Agentic Workflow Infra	Merged
46875	fix: propagate tools.startup-timeout to gateway.startupTimeout (prevents safeoutputs eviction)	Agentic Workflow Infra	Merged
46886	docs(spdd): tighten normative language, add safeguards, expand glossary	Agentic Workflow Infra	Closed
46850	fix: nest issue-intent metadata correctly in close_issue and assign_to_agent payloads	Bug Fixes, Tests & CI	Merged
46856	Recognize global.getOctokit() aliases in no-github-request-interpolated-route	Bug Fixes, Tests & CI	Merged
46857	Clarify intent-aware safe-output mutation contracts across optional and strict modes	Bug Fixes, Tests & CI	Closed
46849	feat(init): make --codespaces optional-value; grant discussions/issues write	CLI & Package Features	Merged
46855	Refactor bootstrap profile runner into focused modules	CLI & Package Features	Merged
46961	Rename CLI firewall PolicyRule to FirewallPolicyRule	CLI & Package Features	Merged
47005	chore: CLI version updates 2026-07-21	Container, MCP & Publishing	Merged
47351	feat: add container_pins to aw.json and document action/container substitutions	Container, MCP & Publishing	Closed
47405	Refresh gh-aw-node packages and multi-platform publishing	Container, MCP & Publishing	Merged
46893	[WIP] Add robots.txt and llms.txt to docs site	AI Engine & Docs	Closed
46928	Switch AI Moderator engine from codex to copilot	AI Engine & Docs	Closed
46986	Add Copilot AI-credits fallback pricing and skip AI-credits guardrails for BYOK	AI Engine & Docs	Closed
47018	Render compressed model alias before AIC in generated footers	Footer/Metadata (Sous Chef)	Merged
47356	Mitigate safe-outputs review failures when inline comment anchors are partially invalid	Footer/Metadata (Sous Chef)	Merged
47374	Add container image substitutions for private registries	Footer/Metadata (Sous Chef)	Merged

(Full 1,000-row dataset available in the workflow's cache-memory artifacts; table above is a representative sample per cluster to keep the report readable.)

Key Findings

1. Infra/publishing tasks (containers, MCP, multi-platform builds) fail to merge far more often than any other category — 56.4% vs. a 74.8% overall average, despite below-average review counts (1.9 avg reviews vs. 3.0 dataset average). Low review intensity paired with low success suggests these PRs may be getting closed without much discussion — possibly abandoned/superseded rather than actively rejected after review.
2. The best-performing cluster (Footer/Metadata, 90% success) is also the most heavily reviewed (12.2 comments, 5.9 reviews avg) — more scrutiny correlates with success here, not the reverse, which argues against "review fatigue" as an explanation for the container/MCP cluster's low rate.
3. Nearly 40% of all copilot PR volume is "Agentic Workflow Infra" work — this repository's copilot usage is dominated by meta-work on the agentic workflow system itself, not general application code.
4. Bug-fix/test/CI PRs (cluster 2, 293 tasks) run a middle-of-the-road but solid 80.2% success rate with above-average review intensity — this is the "bread and butter" category and it's working well.

Recommendations

1. Investigate the Container/MCP/Publishing cluster (56.4% success): pull a sample of its 34 closed-without-merge PRs and check whether they were closed for scope/spec reasons (task under-specified) or technical failures (can't validate multi-arch builds/registries locally). If it's the former, tightening prompt templates for this task category could meaningfully raise the repo-wide success rate given its outsized average diff (35.2 files).
2. Consider whether "Agentic Workflow Infra" (379 tasks, largest cluster) should be split into sub-categories in a future analysis — at nearly 38% of all volume it's likely masking meaningfully different sub-patterns (e.g., workflow YAML edits vs. copilot/gh-aw engine internals) that a higher k or a second-pass clustering on just this cluster's PRs would separate.
3. Wire up workflow-log ingestion (turns/cost) for the next run — this analysis lacked turn-count and token-cost data. Adding gh-aw logs ingestion, even sampled (e.g., 1-in-5 PRs) rather than exhaustive, would let future reports correlate cluster success rate with agent iteration count and cost, which is currently a blind spot.

---

Generated by Prompt Clustering Analysis (Run: 31093718214)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · agent · 119.1 AIC · ⊞ 13.7K · ◷

• expires on Aug 7, 2026, 2:56 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #51051.