[prompt-analysis] Copilot PR Prompt Analysis - 2026-08-06

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-08-06

Summary

Analysis Period: Last 30 days
Total PRs: 1000 | Merged: 752 (75.2%) | Closed: 245 (24.5%) | Open: 3 (0.3%)

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Testing	50	42	84.0%
Other	90	71	78.9%
Documentation	34	26	76.5%
Feature Addition	143	108	75.5%
Bug Fix	640	478	74.7%
Refactoring	43	27	62.8%

Categorized by keyword matching on title+body (fix/bug/error -> Bug Fix; add/implement/create -> Feature Addition; refactor/optimize -> Refactoring; document/docs -> Documentation; test/coverage -> Testing).

Prompt Analysis

Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 235 words (n=752)
• Most common keywords: workflow, actions, engine, agent, comment, generated
• Structure: nearly all use a ## Summary section with bullet points describing concrete changes, often followed by a ## Validation section listing test/build commands run

Example successful prompts:

1. PR test: isolate update checks from agent CI environment #50918: "## Summary - clear COPILOT_AGENT_SESSION_ID in update-check tests that require a non-CI environment..." -> Merged
2. PR Fix Copilot CLI path in AWF sandboxes #50908: "## Summary - resolve the activated Copilot CLI from PATH after installation, stage it in RUNNER_TEMP..." -> Merged
3. PR fix: move poutine:ignore suppression comments inside run: script bodies #50902: "Poutine's untrusted_checkout_exec rule was still firing on 10 already-reviewed steps across smoke-workflow-call.lock.yml..." -> Merged

Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 338 words (n=245) — noticeably longer than merged prompts, often padded with quoted issue context rather than concrete change descriptions
• [WIP]-prefixed titles: 107 PRs, only 36.4% merged vs 79.8% for non-WIP titles — a >2x gap
• Placeholder/boilerplate bodies starting with "Thanks for asking me to work on this. I will get started on it..." (auto-generated stub, not yet replaced with real content): 53 PRs, only 18.9% merged vs 78.4% for PRs with substantive bodies
• Keywords skewed toward pasted security-scan noise: redacted, debian, securitytracker, lgpl — suggesting some closed PRs originated from vulnerability-scan/dependency-audit prompts that stalled before a plan was finalized

Example unsuccessful prompts:

1. PR [WIP] Enable and upload structured JSONL run log on external path #50913: "[WIP] Enable and upload structured JSONL run log on external path" — body: "Thanks for asking me to work on this. I will get started on it..." -> Closed
2. PR [WIP] Fix push_to_pull_request_branch for HEAD-only bundles #50912: "[WIP] Fix push_to_pull_request_branch for HEAD-only bundles" — same boilerplate stub -> Closed
3. PR [WIP] Fix workflows to be Zizmor compliant in version 0.85.4 #50911: "[WIP] Fix workflows to be Zizmor compliant in version 0.85.4" — same boilerplate stub -> Closed

Key Insights

• Pattern 1: PRs left in [WIP] state with the boilerplate "Thanks for asking me to work on this..." stub body are the strongest predictor of failure — only 18.9% merge rate vs 78.4% for PRs with a real, filled-in description. These represent tasks that were started but abandoned before Copilot produced a concrete plan/diff summary.
• Pattern 2: Prompts leading to merged PRs are shorter and more structured (avg 235 words, using ## Summary + ## Validation sections with concrete bullet points) than closed-PR prompts (avg 338 words, often re-pasting long issue/scan text without a clear actionable summary).
• Pattern 3: Testing-labeled prompts have the highest success rate (84.0%), while Refactoring prompts lag (62.8%) — broad, less-scoped asks like "refactor"/"optimize" correlate with more back-and-forth and closures than narrowly-scoped test or bug-fix requests.

Recommendations

Based on today's analysis:

1. DO: Write prompts with a concise ## Summary (bulleted, concrete changes) and a ## Validation section naming the exact test/build commands to run — this pattern dominates merged PRs.
2. DO: Scope refactoring/optimization requests narrowly (specific file/function) rather than broad "improve X" asks, to lift the current 62.8% success rate closer to bug-fix/testing rates.
3. AVOID: Leaving PRs as [WIP] with the default "Thanks for asking me to work on this..." placeholder body — these are essentially abandoned drafts with an 18.9% merge rate. Either complete the task description promptly or close/retry with a clearer, complete prompt from the start.

Data Quality Notes

• All 1000 PRs in the dataset had non-empty bodies (0 missing).
• Sample size (1000 PRs, 245 closed) is sufficient for statistically meaningful category comparisons.

Historical Trends

This is the first recorded analysis — no prior history was available in cache-memory for trend comparison. Today's results have been saved as the baseline for future daily comparisons.

Date	PRs	Success Rate	Top Category
2026-08-06	1000	75.2%	Testing (84.0%)

---

Generated by Copilot PR Prompt Analysis (Run: §31128917942)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · auto · 28.2 AIC · ⊞ 10.8K · ◷

• expires on Aug 7, 2026, 2:27 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-07T22:27:03.917Z.

Closed by Workflow