[feature not live] docs(flaky-tests): document AI Investigation tab#539
Draft
samgutentag wants to merge 5 commits intomainfrom
Draft
[feature not live] docs(flaky-tests): document AI Investigation tab#539samgutentag wants to merge 5 commits intomainfrom
samgutentag wants to merge 5 commits intomainfrom
Conversation
Adds a new page covering the Analysis tab introduced in trunk-io/trunk2#3504. Documents how to view AI investigation results, trigger new investigations, apply fix recommendations, and browse investigation history. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
Contributor
|
Claude finished @samgutentag's task in 1m 39s —— View job PR Review
Overall the page is well-structured, follows the repo's documentation conventions, and all relative links resolve to existing files. Two issues found: Blocking
Non-blocking
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Contributor
|
Claude finished @samgutentag's task in 1m 38s —— View job PR Review
Overall the page is well-structured and follows the repo's documentation conventions. Three issues found — two blocking, one non-blocking. Blocking
Non-blocking
summary.md — Entry placement and link look correct. |
Member
Author
|
Triage status (2026-05-05): up to date with main. Needs review before merge. |
Member
Author
|
Verification status (2026-05-08): Eng merged but flag off in prod. Hold off.
Suggested next action: re-run |
Resolves additive conflict in summary.md by keeping both Test Collections (added in main since this branch opened) and the new AI Investigation entry from this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Confidence band ranges made mutually exclusive (50-79% for yellow). Source: fact-type-config.ts:58-66 (>= 80 returns green, then >= 50 returns yellow, else orange). - "CI Logs" fact type description rewritten so it reads as analysis input rather than as supplementing test output. - Fixed broken link: get-root-cause-analysis.md was renamed to fix-flaky-test.md. - Apply Fix button is rendered unconditionally when an investigation exists (analysis-tab.tsx:275-282); removed the "if produced actionable findings" qualifier and reordered to match UI button order. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two gaps in the original draft surfaced when reading analysis-tab.tsx end to end: - Empty states: the tab has two distinct empty states. The first shows when aiFlakyInvestigationEnabled is DISABLED at the repo level (no Analyze button, "Go to Settings" link instead). The second shows when the setting is enabled but no investigation has run yet. Source: analysis-tab.tsx:87-160. - Summary section: the tab renders the AI's prose summary of the investigation between the Latest Analysis header and Key Findings. Source: analysis-tab.tsx:295-299. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
Claude finished @samgutentag's task in 2m 30s —— View job PR Review
Page is well-structured, follows the page format used elsewhere in
I verified the four relative links in the file all resolve:
Non-blocking grammar/style nits (inline suggestions posted):
No blocking issues. Note that triage status (per Sam's comment) is still |
| Trunk Flaky Tests can automatically investigate a flaky test and surface root cause findings directly in the Trunk web app. The **Analysis** tab on a test detail page shows the latest investigation results, lets you trigger a new investigation, apply a fix, or browse past investigations. | ||
|
|
||
| {% hint style="info" %} | ||
| The Analysis tab requires a GitHub app installation for the repository. If your repo does not have the Trunk GitHub app installed, the Analyze button will be disabled. |
Contributor
There was a problem hiding this comment.
Style nit: elsewhere in this page UI labels are bolded (e.g. **Analyze** button on line 38, Analyze again on line 26 unbolded). For consistency with the bolded usages, consider:
Suggested change
| The Analysis tab requires a GitHub app installation for the repository. If your repo does not have the Trunk GitHub app installed, the Analyze button will be disabled. | |
| The Analysis tab requires a GitHub app installation for the repository. If your repo does not have the Trunk GitHub app installed, the **Analyze** button will be disabled. |
|
|
||
| The tab shows an empty state in two cases: | ||
|
|
||
| - **AI investigation disabled for the repo**: If `aiFlakyInvestigationEnabled` is set to `DISABLED` in the repo's Flaky Tests settings, the tab shows a message that AI investigation is not enabled along with a **Go to Settings** button. No Analyze button is shown until the setting is enabled. |
Contributor
There was a problem hiding this comment.
Minor readability nit: the "along with a button" phrase modifies what the tab shows, but as written it reads as if it's modifying "not enabled". A comma disambiguates:
Suggested change
| - **AI investigation disabled for the repo**: If `aiFlakyInvestigationEnabled` is set to `DISABLED` in the repo's Flaky Tests settings, the tab shows a message that AI investigation is not enabled along with a **Go to Settings** button. No Analyze button is shown until the setting is enabled. | |
| - **AI investigation disabled for the repo**: If `aiFlakyInvestigationEnabled` is set to `DISABLED` in the repo's Flaky Tests settings, the tab shows a message that AI investigation is not enabled, along with a **Go to Settings** button. No Analyze button is shown until the setting is enabled. |
|
|
||
| At the top, you will see: | ||
|
|
||
| - The **overall confidence score** (color-coded green for 80% and above, yellow for 50% to 79%, orange below 50%) |
Contributor
There was a problem hiding this comment.
Parallelism nit: the third clause drops "for" — "green for 80%…, yellow for 50%…, orange below 50%". Adding "for" keeps the three bands parallel:
Suggested change
| - The **overall confidence score** (color-coded green for 80% and above, yellow for 50% to 79%, orange below 50%) | |
| - The **overall confidence score** (color-coded green for 80% and above, yellow for 50% to 79%, orange for below 50%) |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
samgutentag
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
flaky-tests/ai-investigation.mddocumenting the Analysis tab on the Flaky Tests test detail page (trunk-io/trunk2#3504)summary.mdSource
Test plan