[build-tools] Fix ai-*.json is not guarenteed in Maestro 2.5#3639
[build-tools] Fix ai-*.json is not guarenteed in Maestro 2.5#3639
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. |
|
Subscribed to pull request
Generated by CodeMention |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3639 +/- ##
==========================================
+ Coverage 56.62% 56.63% +0.01%
==========================================
Files 886 888 +2
Lines 38331 38416 +85
Branches 7977 8009 +32
==========================================
+ Hits 21703 21754 +51
- Misses 16530 16564 +34
Partials 98 98 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
3d35c3e to
a13b158
Compare
c0d875b to
0c794e5
Compare
0c794e5 to
64b5fa4
Compare
10cf1db to
d439776
Compare
64b5fa4 to
08a9151
Compare
060eefd to
52a58b4
Compare
08a9151 to
11f2a04
Compare
52a58b4 to
e6705c9
Compare
aa0cf30 to
e222a62
Compare
7448f87 to
be9bd1a
Compare
e222a62 to
14e6591
Compare
sjchmiela
left a comment
There was a problem hiding this comment.
a bit hard to wrap one's head around everything that's needed, but i trust you and your testing. i wish the junit output had everything we need. maybe we can submit a pull request to maestro?
| import path from 'path'; | ||
| import yaml from 'yaml'; | ||
|
|
||
| const YAML_EXT = /\.ya?ml$/i; |
There was a problem hiding this comment.
nit: we can do manual === || === check which i think is slightly easier
There was a problem hiding this comment.
I am leaning to keep this as this is simple enough and also handles uppercase/lowercase.
| } | ||
| } | ||
|
|
||
| async function discoverFlows({ |
There was a problem hiding this comment.
what is the missing piece of information from maestro that could let us not have this? maybe they'd consider adding filepath to junit test results? i think this would fix the problem for us too?
There was a problem hiding this comment.
Yeah that's actually a great idea.
Signed-off-by: Ash Wu <hsatac@gmail.com>
be9bd1a to
7f3c9ce
Compare
Signed-off-by: Ash Wu <hsatac@gmail.com>
|
⏩ The changelog entry check has been skipped since the "no changelog" label is present. |

Why
Maestro 2.5.0 (released 2026-04-27) changed
TestDebugReporter.saveSuggestionsto early-return when a flow has no AI screen outputs. As a result,ai-${flow_name}.jsonis now only emitted for flows using AI commands (e.g.assertWithAI).Two callers in
@expo/build-toolsrelied onai-*.jsonfor theflow_name → flow_file_pathmapping:parseFailedFlowsFromJUnit): lookup misses every time on 2.5.0+, so smart retry silently degrades to dumb retry — a functional regression of [build-tools] Add eas/run_maestro_tests step with smart retry #3635 against the latest Maestro.parseMaestroResults): thepathon uploaded testCaseResults falls back to flow name. Cosmetic-only —pathis read in the workflow detail UI as a hover tooltip.Linear: ENG-19927. Stacked on #3635.
How
Replace the
ai-*.jsonwalk with a self-builtflow_name → flow_file_pathmap (buildFlowNameToPathMap). The helper walks the user'sflow_pathsinput, parses each flow YAML's top-levelname(or falls back to file basename without extension), and produces aname → pathmap. BothparseFailedFlowsFromJUnitandparseMaestroResultsconsume it.pathsemantics).nulland fall back to the previous degraded behavior.flow_pathsinput toeas/report_maestro_test_results. Universe-side wiring (passingjob.params.flow_pathinto this input) is a separate PR.Behavior change for
eas/report_maestro_test_results: single-attemptretryCountis no longer derived fromai-*.jsonoccurrence count, falls back to0. Production impact: zero — no readers ofretryCount, and the only caller of the single-attempt branch (the bash step) is retired by #3641.Backward compatibility during deployment: between this PR landing and the universe wiring landing, old job specs (without
flow_pathson the report step) run against the new code. The step silently falls back tonullmap → flow-name-as-path, no warnings.Test Plan
maestroFlowDiscovery.test.tscovering: top-level name, basename fallback, malformed YAML, missing files, leading---document marker, directory walks, symlink cycles (real fs), workspaceconfig.yamlskip, duplicate-name handling, same-file dedup, absolute paths.maestroResultParser.test.ts,reportMaestroTestResults.test.ts,runMaestroTests.test.ts).--shard-split=2: failed flows from attempt 1 retried with project-relative paths; sharding adjusted to the retried subset.