-
Notifications
You must be signed in to change notification settings - Fork 13
[feature not live] docs(flaky-tests): document AI Investigation tab #539
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
953b4aa
16d3f08
8b52f78
af9a158
9cf8ddd
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,102 @@ | ||||||
| --- | ||||||
| description: >- | ||||||
| Use the Analysis tab on a test detail page to view AI-generated investigation | ||||||
| results, trigger new investigations, and get fix recommendations. | ||||||
| --- | ||||||
|
|
||||||
| # AI Investigation | ||||||
|
|
||||||
| Trunk Flaky Tests can automatically investigate a flaky test and surface root cause findings directly in the Trunk web app. The **Analysis** tab on a test detail page shows the latest investigation results, lets you trigger a new investigation, apply a fix, or browse past investigations. | ||||||
|
|
||||||
| {% hint style="info" %} | ||||||
| The Analysis tab requires a GitHub app installation for the repository. If your repo does not have the Trunk GitHub app installed, the Analyze button will be disabled. | ||||||
| {% endhint %} | ||||||
|
|
||||||
| ## Open the Analysis tab | ||||||
|
|
||||||
| 1. Navigate to **Flaky Tests** in the Trunk web app. | ||||||
| 2. Click a test case to open the test detail page. | ||||||
| 3. Select the **Analysis** tab. | ||||||
|
|
||||||
| ### Empty states | ||||||
|
|
||||||
| The tab shows an empty state in two cases: | ||||||
|
|
||||||
| - **AI investigation disabled for the repo**: If `aiFlakyInvestigationEnabled` is set to `DISABLED` in the repo's Flaky Tests settings, the tab shows a message that AI investigation is not enabled along with a **Go to Settings** button. No Analyze button is shown until the setting is enabled. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Minor readability nit: the "along with a button" phrase modifies what the tab shows, but as written it reads as if it's modifying "not enabled". A comma disambiguates:
Suggested change
|
||||||
| - **Investigation enabled, no analysis yet**: If the setting is enabled but no investigation has run for this test yet, the tab shows an **Analyze** button. When the GitHub app is installed, tests are also analyzed automatically when they first become flaky. | ||||||
|
|
||||||
| ## Understanding investigation results | ||||||
|
|
||||||
| When an investigation is available, the Analysis tab shows: | ||||||
|
|
||||||
| ### Latest Analysis header | ||||||
|
|
||||||
| At the top, you will see: | ||||||
|
|
||||||
| - The **overall confidence score** (color-coded green for 80% and above, yellow for 50% to 79%, orange below 50%) | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Parallelism nit: the third clause drops "for" — "green for 80%…, yellow for 50%…, orange below 50%". Adding "for" keeps the three bands parallel:
Suggested change
|
||||||
| - A relative timestamp for when the investigation ran | ||||||
| - An **Analyze** button to trigger a new investigation | ||||||
| - An **Apply Fix** button to open the fix options modal | ||||||
| - A **History** button to view past investigations | ||||||
|
|
||||||
| ### Summary | ||||||
|
|
||||||
| Below the header, the Analysis tab renders the AI's prose summary of the investigation. The summary highlights the most likely root cause, references specific findings, and links to relevant files, lines, and commits. Use it as the entry point to the investigation before drilling into Key Findings. | ||||||
|
|
||||||
| ### Key Findings | ||||||
|
|
||||||
| The Key Findings section lists the top three findings ordered by impact. Each finding shows: | ||||||
|
|
||||||
| | Field | Description | | ||||||
| |---|---| | ||||||
| | Fact type badge | The analysis source that produced the finding | | ||||||
| | Confidence percentage | How confident the AI is in this specific finding | | ||||||
| | Finding content | A summary with links to relevant code, CI logs, or commits | | ||||||
|
|
||||||
| A collapsible **Other Findings** section holds any additional findings beyond the top three. | ||||||
|
|
||||||
| ### Fact types | ||||||
|
|
||||||
| Each finding is labeled with the analysis source used to produce it: | ||||||
|
|
||||||
| | Fact type | What it analyzes | | ||||||
| |---|---| | ||||||
| | **CI Logs** | CI workflow logs analyzed alongside the test's failure output | | ||||||
| | **Git Blame** | Recent code changes that may have introduced flakiness | | ||||||
| | **Failure Mode** | Patterns in error outputs and failure metadata | | ||||||
| | **Test Purpose** | What the test covers and how it has drifted from its intent | | ||||||
| | **Environment** | Environmental factors such as resource constraints or timing | | ||||||
| | **Co-failure** | Other tests that fail at the same time, pointing to shared causes | | ||||||
| | **File Co-change** | Related files that have changed alongside the failing test | | ||||||
|
|
||||||
| ## Trigger a new investigation | ||||||
|
|
||||||
| Click **Analyze** to open the Trigger Analysis modal. Click **Run Analysis** to kick off a fresh investigation. Trunk will analyze the test using the latest CI logs, git history, and failure data. | ||||||
|
|
||||||
| Investigations run automatically when a test first becomes flaky (if the GitHub app is installed). You can trigger additional investigations manually at any time. | ||||||
|
|
||||||
| ## Apply a fix | ||||||
|
|
||||||
| Click **Apply Fix** to open the Apply Fix modal. Trunk surfaces the fix options available for the current investigation: | ||||||
|
|
||||||
| - **Copy Prompt** — copies a prompt you can paste into an AI coding assistant to guide it toward the fix | ||||||
| - **Fix with MCP** — connects directly to the Trunk MCP server to apply the fix (requires the MCP server to be configured) | ||||||
| - **Automate with Webhooks** — links to the webhooks configuration to automate fix workflows | ||||||
|
|
||||||
| See [Use MCP Server](use-mcp-server/README.md) for information on setting up the MCP integration. | ||||||
|
|
||||||
| ## View investigation history | ||||||
|
|
||||||
| Click **History** to open the Analysis History modal. This shows past investigations for the test case, each with: | ||||||
|
|
||||||
| - A confidence badge | ||||||
| - A findings count | ||||||
| - A timestamp | ||||||
|
|
||||||
| Click any past investigation to expand its details. | ||||||
|
|
||||||
| ## Related | ||||||
|
|
||||||
| - [Fix Flaky Test (MCP)](use-mcp-server/mcp-tool-reference/fix-flaky-test.md) | ||||||
| - [Webhooks](webhooks/README.md) | ||||||
| - [Managing detected flaky tests](managing-detected-flaky-tests.md) | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Style nit: elsewhere in this page UI labels are bolded (e.g.
**Analyze**button on line 38,Analyzeagain on line 26 unbolded). For consistency with the bolded usages, consider: