Skip to content

[feature not live] docs(flaky-tests): document test_case.investigation_completed webhook event#558

Draft
samgutentag wants to merge 5 commits intomainfrom
sam-gutentag/investigation-completed-webhook-event
Draft

[feature not live] docs(flaky-tests): document test_case.investigation_completed webhook event#558
samgutentag wants to merge 5 commits intomainfrom
sam-gutentag/investigation-completed-webhook-event

Conversation

@samgutentag
Copy link
Copy Markdown
Member

Summary

  • Adds a new reference page for the test_case.investigation_completed webhook event, documenting the full payload schema including markdown_summary, facts[], repository, and test_case fields
  • Updates the webhooks README to link to the new page
  • Adds the page to summary.md (table of contents)

Context

Shipped in v156 (trunk-io/trunk2#3592): the markdown_summary field and facts[].content in the investigation_completed webhook payload now contain rendered citation links (actual URLs) instead of raw CITE:TAG placeholders. This makes the event payload directly usable for Slack messages, ticket creation, etc.

Previously there was no docs page for this event at all.

Test plan

  • Verify new page renders correctly in GitBook preview
  • Check all internal links resolve (investigation-completed-event.md, README.md, summary.md)
  • Confirm example JSON payload fields match the Zod schema in trunk2

https://claude.ai/code/session_01B2TQ276xj88AoXyrgaahah

… event

The investigation_completed webhook event now delivers rendered citation
links in markdown_summary and facts[].content (shipped in v156). There was
no existing docs page for this event.

Shipped in v156 via trunk-io/trunk2#3592.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

https://claude.ai/code/session_01B2TQ276xj88AoXyrgaahah
@trunk-staging-io
Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@samgutentag
Copy link
Copy Markdown
Member Author

Triage status (2026-05-05): merge conflicts with main in summary.md and flaky-tests/webhooks/README.md. Resolve before review.

@samgutentag
Copy link
Copy Markdown
Member Author

samgutentag commented May 6, 2026

Verification status (2026-05-07): pending

Eng merged but flag off in prod. Hold off.

  • Eng PR: trunk-io/trunk2#3592 (merged 2026-04-17, stacked merge of #3577/#3578/#3579)
  • Flag: emitInvestigationResultWebhooks ("Flaky Tests Emit Investigate Event" in LaunchDarkly)
  • Signals:
    • Merge commit on main (status: behind, merge intact)
    • Flag still gated in trunk2 main (ts/packages/ai/common/flags.ts, ts/apps/backend/flaky-tests-investigation-poller/src/webhook.ts, and the corresponding vitest all reference EMIT_INVESTIGATION_RESULT_WEBHOOKS_FLAG)
    • LaunchDarkly bot updated the flag in #production-notifications on 2026-04-23 and 2026-04-27 (after mergedAt). Active management, not a 100% rollout signal.
    • Tyler Jang in #team-flaky-tests on 2026-04-27 listed this flag under AI Flaky Tests Investigation rollout context: "Still awaiting feedback from Brex and Chainlink... Laying the ground work to run Descript, Gusto/web, Caseware in stealth." Feature is in stealth with a limited customer set, not broadly available.
    • e2e flag default is null in ts/apps/e2e/flags.json
    • No follow-up "delete legacy" or "100% rollout" PR found in trunk2

Suggested next action: re-run /verify-docs-pr 558 after the flag flips to 100% in production. @tyler-jang or #team-flaky-tests can confirm the rollout schedule.

samgutentag and others added 4 commits May 6, 2026 20:55
…ation-completed-webhook-event

# Conflicts:
#	flaky-tests/webhooks/README.md
…lassname casing

- Remove the invented facts[].impact field from the example payload and
  the field reference table. The Zod payload schema in trunk2 only
  defines fact_type, content, and confidence on each fact.
- Fix vale flag on the classname row description.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@samgutentag samgutentag changed the title docs(flaky-tests): document test_case.investigation_completed webhook event [feature not live] docs(flaky-tests): document test_case.investigation_completed webhook event May 7, 2026
@claude
Copy link
Copy Markdown
Contributor

claude Bot commented May 7, 2026

Claude finished @samgutentag's task in 1m 36s —— View job


Review

  • Read changed files (3 files: new page + README + summary.md)
  • Compare new page against sibling webhook pages and the README's existing event reference
  • Post inline comments for issues found

Findings

Blocking

  • flaky-tests/webhooks/investigation-completed-event.md:127 — a stray Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> trailer escaped from a commit message into the page body and will render as visible text at the bottom of the published doc. Inline suggestion posted.

Should fix before merge

  • flaky-tests/webhooks/investigation-completed-event.md:89 vs flaky-tests/webhooks/README.md:73fact_type examples disagree (CI_LOGS/COMMIT/TEST_RUN on the new page, GIT_BLAME in the README). Reconcile against the actual Zod enum in trunk2 so the two pages don't contradict each other.

Nits

  • Heading case in the new page is sentence-case (Event type, Markdown summary, Enabling the webhook) while sibling pages use Title Case section headings. Worth aligning.
  • summary.md:122 TOC entry AI Investigation Completed Event uses Title Case while other webhooks entries are sentence case (Linear integration, Jira integration). Suggested rewrite inline.

Looks good

  • Frontmatter description follows the same shape as siblings.
  • New page is added to both summary.md and README.md, and internal links resolve (README.md, summary.md).
  • The example JSON, field tables, and Slack snippet all line up with each other (top-level fields, repository, test_case, facts[]).
  • The PR description correctly flags this is gated on the emitInvestigationResultWebhooks rollout — appropriate to hold merge until verification flips to ready, per Sam's 2026-05-07 triage note.

review_passed: false (blocked on the stray trailer at line 127; the README/new-page fact_type mismatch should also be resolved)
· sam-gutentag/investigation-completed-webhook-event


For general webhook setup, see [Webhooks](README.md).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: A Co-Authored-By commit-message trailer was accidentally committed into the page body. It will render as visible text at the bottom of the published doc. Belongs in the commit message, not the file.

Suggested change
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
For general webhook setup, see [Webhooks](README.md).

Fix this →


| Field | Type | Description |
|---|---|---|
| `fact_type` | string | Category of evidence (e.g., `CI_LOGS`, `COMMIT`, `TEST_RUN`). |
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact_type examples here (CI_LOGS, COMMIT, TEST_RUN) don't match the example in README.md line 73, which uses GIT_BLAME. Worth reconciling so a reader scanning both pages sees consistent values — and ideally the list reflects the actual enum from the trunk2 schema.


When Trunk's AI finishes investigating a flaky test, it fires a `test_case.investigation_completed` webhook event. Your endpoint receives the investigation findings, a confidence score, and a rendered markdown summary you can post to Slack, create tickets from, or feed into your own tooling.

## Event type
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this page uses sentence-case H2 headings (Event type, Markdown summary, Enabling the webhook), while sibling pages (README.md, slack-integration.md) use Title Case for section headings (Supported Events, Configure Slack webhooks). Worth aligning to keep the section nav consistent across the webhooks folder.

Comment thread summary.md
* [GitHub Issues integration](flaky-tests/webhooks/github-issues-integration.md)
* [Linear integration](flaky-tests/webhooks/linear-integration.md)
* [Jira integration](flaky-tests/webhooks/jira-integration.md)
* [AI Investigation Completed Event](flaky-tests/webhooks/investigation-completed-event.md)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the surrounding entries use sentence case (Linear integration, Jira integration, Microsoft Teams integration). Consider AI investigation completed event for consistency.

Suggested change
* [AI Investigation Completed Event](flaky-tests/webhooks/investigation-completed-event.md)
* [AI investigation completed event](flaky-tests/webhooks/investigation-completed-event.md)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants