Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 102 additions & 0 deletions flaky-tests/ai-investigation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
---
description: >-
Use the Analysis tab on a test detail page to view AI-generated investigation
results, trigger new investigations, and get fix recommendations.
---

# AI Investigation

Trunk Flaky Tests can automatically investigate a flaky test and surface root cause findings directly in the Trunk web app. The **Analysis** tab on a test detail page shows the latest investigation results, lets you trigger a new investigation, apply a fix, or browse past investigations.

{% hint style="info" %}
The Analysis tab requires a GitHub app installation for the repository. If your repo does not have the Trunk GitHub app installed, the Analyze button will be disabled.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style nit: elsewhere in this page UI labels are bolded (e.g. **Analyze** button on line 38, Analyze again on line 26 unbolded). For consistency with the bolded usages, consider:

Suggested change
The Analysis tab requires a GitHub app installation for the repository. If your repo does not have the Trunk GitHub app installed, the Analyze button will be disabled.
The Analysis tab requires a GitHub app installation for the repository. If your repo does not have the Trunk GitHub app installed, the **Analyze** button will be disabled.

{% endhint %}

## Open the Analysis tab

1. Navigate to **Flaky Tests** in the Trunk web app.
2. Click a test case to open the test detail page.
3. Select the **Analysis** tab.

### Empty states

The tab shows an empty state in two cases:

- **AI investigation disabled for the repo**: If `aiFlakyInvestigationEnabled` is set to `DISABLED` in the repo's Flaky Tests settings, the tab shows a message that AI investigation is not enabled along with a **Go to Settings** button. No Analyze button is shown until the setting is enabled.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor readability nit: the "along with a button" phrase modifies what the tab shows, but as written it reads as if it's modifying "not enabled". A comma disambiguates:

Suggested change
- **AI investigation disabled for the repo**: If `aiFlakyInvestigationEnabled` is set to `DISABLED` in the repo's Flaky Tests settings, the tab shows a message that AI investigation is not enabled along with a **Go to Settings** button. No Analyze button is shown until the setting is enabled.
- **AI investigation disabled for the repo**: If `aiFlakyInvestigationEnabled` is set to `DISABLED` in the repo's Flaky Tests settings, the tab shows a message that AI investigation is not enabled, along with a **Go to Settings** button. No Analyze button is shown until the setting is enabled.

- **Investigation enabled, no analysis yet**: If the setting is enabled but no investigation has run for this test yet, the tab shows an **Analyze** button. When the GitHub app is installed, tests are also analyzed automatically when they first become flaky.

## Understanding investigation results

When an investigation is available, the Analysis tab shows:

### Latest Analysis header

At the top, you will see:

- The **overall confidence score** (color-coded green for 80% and above, yellow for 50% to 79%, orange below 50%)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parallelism nit: the third clause drops "for" — "green for 80%…, yellow for 50%…, orange below 50%". Adding "for" keeps the three bands parallel:

Suggested change
- The **overall confidence score** (color-coded green for 80% and above, yellow for 50% to 79%, orange below 50%)
- The **overall confidence score** (color-coded green for 80% and above, yellow for 50% to 79%, orange for below 50%)

- A relative timestamp for when the investigation ran
- An **Analyze** button to trigger a new investigation
- An **Apply Fix** button to open the fix options modal
- A **History** button to view past investigations

### Summary

Below the header, the Analysis tab renders the AI's prose summary of the investigation. The summary highlights the most likely root cause, references specific findings, and links to relevant files, lines, and commits. Use it as the entry point to the investigation before drilling into Key Findings.

### Key Findings

The Key Findings section lists the top three findings ordered by impact. Each finding shows:

| Field | Description |
|---|---|
| Fact type badge | The analysis source that produced the finding |
| Confidence percentage | How confident the AI is in this specific finding |
| Finding content | A summary with links to relevant code, CI logs, or commits |

A collapsible **Other Findings** section holds any additional findings beyond the top three.

### Fact types

Each finding is labeled with the analysis source used to produce it:

| Fact type | What it analyzes |
|---|---|
| **CI Logs** | CI workflow logs analyzed alongside the test's failure output |
| **Git Blame** | Recent code changes that may have introduced flakiness |
| **Failure Mode** | Patterns in error outputs and failure metadata |
| **Test Purpose** | What the test covers and how it has drifted from its intent |
| **Environment** | Environmental factors such as resource constraints or timing |
| **Co-failure** | Other tests that fail at the same time, pointing to shared causes |
| **File Co-change** | Related files that have changed alongside the failing test |

## Trigger a new investigation

Click **Analyze** to open the Trigger Analysis modal. Click **Run Analysis** to kick off a fresh investigation. Trunk will analyze the test using the latest CI logs, git history, and failure data.

Investigations run automatically when a test first becomes flaky (if the GitHub app is installed). You can trigger additional investigations manually at any time.

## Apply a fix

Click **Apply Fix** to open the Apply Fix modal. Trunk surfaces the fix options available for the current investigation:

- **Copy Prompt** — copies a prompt you can paste into an AI coding assistant to guide it toward the fix
- **Fix with MCP** — connects directly to the Trunk MCP server to apply the fix (requires the MCP server to be configured)
- **Automate with Webhooks** — links to the webhooks configuration to automate fix workflows

See [Use MCP Server](use-mcp-server/README.md) for information on setting up the MCP integration.

## View investigation history

Click **History** to open the Analysis History modal. This shows past investigations for the test case, each with:

- A confidence badge
- A findings count
- A timestamp

Click any past investigation to expand its details.

## Related

- [Fix Flaky Test (MCP)](use-mcp-server/mcp-tool-reference/fix-flaky-test.md)
- [Webhooks](webhooks/README.md)
- [Managing detected flaky tests](managing-detected-flaky-tests.md)
1 change: 1 addition & 0 deletions summary.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@
* [Multiple Repositories and Forks](flaky-tests/get-started/multiple-repositories.md)
* [Managing detected flaky tests](flaky-tests/managing-detected-flaky-tests.md)
* [Test Collections](flaky-tests/test-collections.md)
* [AI Investigation](flaky-tests/ai-investigation.md)
* [Dashboard](flaky-tests/dashboard.md)
* [Flaky test detection](flaky-tests/detection/README.md)
* [Pass-on-Retry Monitor](flaky-tests/detection/pass-on-retry-monitor.md)
Expand Down
Loading