diff --git a/flaky-tests/ai-investigation.md b/flaky-tests/ai-investigation.md new file mode 100644 index 00000000..479faeec --- /dev/null +++ b/flaky-tests/ai-investigation.md @@ -0,0 +1,102 @@ +--- +description: >- + Use the Analysis tab on a test detail page to view AI-generated investigation + results, trigger new investigations, and get fix recommendations. +--- + +# AI Investigation + +Trunk Flaky Tests can automatically investigate a flaky test and surface root cause findings directly in the Trunk web app. The **Analysis** tab on a test detail page shows the latest investigation results, lets you trigger a new investigation, apply a fix, or browse past investigations. + +{% hint style="info" %} +The Analysis tab requires a GitHub app installation for the repository. If your repo does not have the Trunk GitHub app installed, the Analyze button will be disabled. +{% endhint %} + +## Open the Analysis tab + +1. Navigate to **Flaky Tests** in the Trunk web app. +2. Click a test case to open the test detail page. +3. Select the **Analysis** tab. + +### Empty states + +The tab shows an empty state in two cases: + +- **AI investigation disabled for the repo**: If `aiFlakyInvestigationEnabled` is set to `DISABLED` in the repo's Flaky Tests settings, the tab shows a message that AI investigation is not enabled along with a **Go to Settings** button. No Analyze button is shown until the setting is enabled. +- **Investigation enabled, no analysis yet**: If the setting is enabled but no investigation has run for this test yet, the tab shows an **Analyze** button. When the GitHub app is installed, tests are also analyzed automatically when they first become flaky. + +## Understanding investigation results + +When an investigation is available, the Analysis tab shows: + +### Latest Analysis header + +At the top, you will see: + +- The **overall confidence score** (color-coded green for 80% and above, yellow for 50% to 79%, orange below 50%) +- A relative timestamp for when the investigation ran +- An **Analyze** button to trigger a new investigation +- An **Apply Fix** button to open the fix options modal +- A **History** button to view past investigations + +### Summary + +Below the header, the Analysis tab renders the AI's prose summary of the investigation. The summary highlights the most likely root cause, references specific findings, and links to relevant files, lines, and commits. Use it as the entry point to the investigation before drilling into Key Findings. + +### Key Findings + +The Key Findings section lists the top three findings ordered by impact. Each finding shows: + +| Field | Description | +|---|---| +| Fact type badge | The analysis source that produced the finding | +| Confidence percentage | How confident the AI is in this specific finding | +| Finding content | A summary with links to relevant code, CI logs, or commits | + +A collapsible **Other Findings** section holds any additional findings beyond the top three. + +### Fact types + +Each finding is labeled with the analysis source used to produce it: + +| Fact type | What it analyzes | +|---|---| +| **CI Logs** | CI workflow logs analyzed alongside the test's failure output | +| **Git Blame** | Recent code changes that may have introduced flakiness | +| **Failure Mode** | Patterns in error outputs and failure metadata | +| **Test Purpose** | What the test covers and how it has drifted from its intent | +| **Environment** | Environmental factors such as resource constraints or timing | +| **Co-failure** | Other tests that fail at the same time, pointing to shared causes | +| **File Co-change** | Related files that have changed alongside the failing test | + +## Trigger a new investigation + +Click **Analyze** to open the Trigger Analysis modal. Click **Run Analysis** to kick off a fresh investigation. Trunk will analyze the test using the latest CI logs, git history, and failure data. + +Investigations run automatically when a test first becomes flaky (if the GitHub app is installed). You can trigger additional investigations manually at any time. + +## Apply a fix + +Click **Apply Fix** to open the Apply Fix modal. Trunk surfaces the fix options available for the current investigation: + +- **Copy Prompt** — copies a prompt you can paste into an AI coding assistant to guide it toward the fix +- **Fix with MCP** — connects directly to the Trunk MCP server to apply the fix (requires the MCP server to be configured) +- **Automate with Webhooks** — links to the webhooks configuration to automate fix workflows + +See [Use MCP Server](use-mcp-server/README.md) for information on setting up the MCP integration. + +## View investigation history + +Click **History** to open the Analysis History modal. This shows past investigations for the test case, each with: + +- A confidence badge +- A findings count +- A timestamp + +Click any past investigation to expand its details. + +## Related + +- [Fix Flaky Test (MCP)](use-mcp-server/mcp-tool-reference/fix-flaky-test.md) +- [Webhooks](webhooks/README.md) +- [Managing detected flaky tests](managing-detected-flaky-tests.md) diff --git a/summary.md b/summary.md index 787916f9..c66b0427 100644 --- a/summary.md +++ b/summary.md @@ -97,6 +97,7 @@ * [Multiple Repositories and Forks](flaky-tests/get-started/multiple-repositories.md) * [Managing detected flaky tests](flaky-tests/managing-detected-flaky-tests.md) * [Test Collections](flaky-tests/test-collections.md) +* [AI Investigation](flaky-tests/ai-investigation.md) * [Dashboard](flaky-tests/dashboard.md) * [Flaky test detection](flaky-tests/detection/README.md) * [Pass-on-Retry Monitor](flaky-tests/detection/pass-on-retry-monitor.md)