Skip to content

ppx_devtools.expect: handle repeated and unreached [%expect]s like ppx_expect - #6

Merged
hhugo merged 16 commits into
mainfrom
expect-unreachable
Nov 16, 2026
Merged

hhugo merged 16 commits into
mainfrom
expect-unreachable

Conversation

@hhugo

@hhugo hhugo commented Nov 3, 2026

Copy link
Copy Markdown
Member

Stacked on #5.

  • Assertion points that run several times with different outputs are no longer a runner error. Outputs are recorded as the test runs and checked when it ends: if they differ, promotion writes all of them under a CR comment, and the test keeps failing until it gives a single output.
  • Unreached assertion points. An [%expect] that never runs (e.g. after an exception) is promoted to [%expect.unreachable], which fails when reached and is then promoted back to an [%expect] holding the output. In a group with [@when] variants, only the default becomes unreachable; the variants keep their snapshots, as the code may run under other axes. [%expect.unreachable] can itself be a [@when] variant.

With no runner errors left in -diff-cmd - mode, every failure goes through dune's diff and promotion, and a test is counted once in the standalone failure count. test/promote/disagree.ml, which checked for the runner error, is folded into promote_me.ml.

🤖 Generated with Claude Code

@hhugo
hhugo force-pushed the expect-uncaught-exn branch from cc888ea to 4b9b01b Compare November 3, 2026 08:16
@hhugo
hhugo force-pushed the expect-unreachable branch from 516f86f to d2ffc3b Compare November 3, 2026 08:16
@hhugo
hhugo force-pushed the expect-uncaught-exn branch from 4b9b01b to cf19038 Compare November 3, 2026 08:42
@hhugo
hhugo force-pushed the expect-unreachable branch 2 times, most recently from a65423d to e183abe Compare November 3, 2026 08:53
@hhugo
hhugo force-pushed the expect-uncaught-exn branch from cf19038 to 713323f Compare November 3, 2026 08:53
Base automatically changed from expect-uncaught-exn to main November 3, 2026 09:00
hhugo and others added 4 commits November 3, 2026 10:02
…x_expect

- An assertion point that runs several times with different outputs is no
  longer a runner error. Promotion records every output under a
  "CR expect_test: Test ran multiple times with different test outputs"
  comment, and the test keeps failing until it gives a single output.
- A plain [%expect] that is never reached is promoted to
  [%expect.unreachable], which is now supported: when reached, it is
  promoted back to an [%expect] holding the output. A group with [@when]
  variants is left alone, as it may only be unreachable under some axes.

Outputs are now recorded as the test runs and checked when it ends. With
no runner errors left, a test can no longer be counted twice in the
failure count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An unreached group with [@when] variants used to be left alone. Now its
active variant becomes an [%expect.unreachable] that keeps the
condition, ([%expect.unreachable] [@when COND]), and the other variants
are left for the axes under which the code may run. When reached, it is
promoted back to [%expect ({| ... |} [@when COND])].

[%expect.unreachable] accordingly takes a [@when] attribute and is part
of the group of the assertion points next to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…achable

When a group with [@when] variants is not reached, its default [%expect]
becomes [%expect.unreachable], and the [@when] variants keep their
snapshots, rather than the active variant becoming a conditional
[%expect.unreachable].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ect]s

- An [%expect] that ran several times listed one output per run under
  its CR, repeats included. As in ppx_expect, it now lists each distinct
  output once, sorted. Only the distinct outputs are kept while the test
  runs.
- An unreached group marked its default [%expect] unreachable even when
  a [@when] variant applied, e.g. ([%expect.unreachable] [@when native]).
  Code reached on some backends only then never converged, each backend's
  promotion undoing the other's. The default is now only marked when it
  is the active variant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hhugo
hhugo force-pushed the expect-unreachable branch from e183abe to 6f606c1 Compare November 3, 2026 09:03
hhugo and others added 12 commits November 4, 2026 09:11
- An [%expect] that saw different outputs passed under dune once its CR
  was promoted, as the corrected file then matched the source. Like
  ppx_expect, the runner now fails until the outputs agree, even under
  dune.
- A reached [%expect.unreachable] with no output is reported as reached,
  rather than as an empty diff.
- [mismatch] is defined with [report], and a [@when] is applied to a
  variant by a single [with_condition] helper.
- [Corrected.Delete] is back, rather than [Replace ""].
- The CR lists each distinct output once, unlike ppx_expect, which lists
  every run: say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An [%expect] that saw different outputs made the runner fail under dune.
Dune only diffs the [.corrected] files when the runner succeeds, so the
CR, and every other correction of the run, could not be promoted.

The runner now fails on such an [%expect] only once there is nothing
left to promote, i.e. when no [.corrected] file differs from its source.
Until then, the diffs fail the test.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The error for an [%expect] that saw different outputs gave the line of
the first variant of the group, rather than of the variant applying in
this environment, which is the one getting the CR.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Adjacent assertion points form a group of variants, of which the plain
[%expect] is the default. A second plain [%expect] was put in the same
group, where it was never selected: it was not checked, not marked
unreachable when unreached, and a second [%expect.unreachable] was not
converted when reached.

A group now has a single default, and a second one starts the next
group. As in ppx_expect, the second of two adjacent [%expect]s is thus
checked against an empty output.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Run without [-diff-cmd -], the runner did not say that an [%expect] saw
different outputs: once the CR was in the source, it only printed a diff
with no change but the indentation.

The error is now printed in both modes, and no correction is made for a
CR that is already in the source, which removes that diff. Under dune,
no [.corrected] file is written in that case, and the runner still
fails.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…he CR

- A second plain [%expect] started a new group, but a [@when] variant
  following a plain [%expect] still joined its group, rather than the
  group of the default after it. The default now ends its group: the
  assertion points following it form the next group.
- The error for different outputs names what gets the CR: the variant
  applying in this environment, which may be an [%expect.unreachable],
  or the [%expect] to be inserted after the group.
- The variant of a group is selected once, rather than twice.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
[check_groups] both dispatched on the outputs of a group and handled
the unreached and different outputs cases, selecting a variant that
[check_group] then matched on again.

It now only dispatches, to [check_unreached] or to [check_reached] (the
former [check_group]), which records the error for different outputs
where it places the CR. No change in behaviour.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
In a group, the first [@when] variant whose condition holds applies,
and otherwise the default, which ends the group. [@when] variants
written after the default were thus silently made a group of their own,
checked against an empty output.

Such variants, with no default of their own, are now an error. The
README states how a variant is selected.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- A test whose body is a lone expression with an attribute, such as
  [print_string "x" [@attr]], did not preprocess ("Attributes not
  allowed here"), and the [%expect] added for its trailing output would
  have been inserted before the attribute, taking it over. The end of
  the body now covers its attributes.
- [when_of_attributes] is defined with [when_condition], rather than
  duplicating it.
- In [check_reached], the flag for different outputs no longer has the
  name of the function recording them.
- The runs with and without [-diff-cmd -] of the test for a promoted CR
  share their expected output, which is the same.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- [[%expect {| ... |}] [@when COND]] was silently an unconditional
  [%expect], while the same form is a conditional variant with
  [%expect.unreachable]. It now is one too, like
  [[%expect ({| ... |} [@when COND])]].
- [Corrected.write] no longer tells whether a corrected file differs
  from its source: every correction changes its source, now that none is
  made for a CR that is already there.
- The function recording a correction is named [record], rather than
  [report] like the one printing in [exit].

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- The literal of a bare [%expect] was inserted one character before the
  end of the node, taken to be its closing bracket. In
  [([%expect] [@when COND])], this is the closing parenthesis, and
  promotion resulted in a syntax error. The literal is now inserted
  after the name of the extension.
- [classify] handles a [@when] following the node in one place for
  [%expect] and [%expect.unreachable].

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- Promoting [{%expect| ... |}] corrupted the source, as the literal of
  this form has no delimiters of its own. The node is now rewritten as
  an [[%expect {| ... |}]].
- A reached [%expect.unreachable] was rewritten together with the
  [@when] following it, dropping any attribute or comment in between.
  Only the name of the node is rewritten now, which keeps what
  surrounds it: [[%expect.unreachable] [@when COND]] becomes
  [[%expect {| ... |}] [@when COND]].
- Groups were numbered in the evaluation order of [::], so the errors
  of a test came out in reverse source order.
- Run without [-diff-cmd -], a reached [%expect.unreachable] was only
  said to be reached when it saw no output.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@hhugo
hhugo merged commit 7776cf9 into main Nov 16, 2026
13 checks passed
@hhugo
hhugo deleted the expect-unreachable branch November 16, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant