Fix score-bounce crash on emoji headings (LC_ALL=C awk) - #40
Conversation
gawk's locale-aware gsub(/[^a-z0-9 ]/) mishandles 4-byte UTF-8
codepoints under a UTF-8 locale: it strips 3 of 4 bytes and leaves
an orphan continuation byte, which makes the downstream `sort` hard
-crash ("Invalid or incomplete multibyte or wide character"). Under
set -e this took down the whole scorer, so report-bounce.sh could
never produce bounce-scores.json for any run whose document had an
emoji in a heading -- surfacing only as the generic
"WARNING: report generation failed" from the co-evolve-bouncer.sh
auto-hook.
Force LC_ALL=C on the three awk extractors (headings/anchors/
markers): they only emit ASCII fingerprints for set comparison, so
byte semantics strip every byte of a multi-byte character cleanly
either way.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6fe70c1c88
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # LC_ALL=C: see extract_headings() above -- same gawk multibyte gsub bug. | ||
| extract_anchors() { | ||
| awk ' | ||
| LC_ALL=C awk ' |
There was a problem hiding this comment.
For runs whose anchor set includes non-ASCII quoted terms, this changes the extractor's normalization semantics: extract_anchors emits quote/code text directly, so under LC_ALL=C tolower() no longer folds characters such as É to é and length() switches to byte counts. A bounce that only lowercases a quoted term like "ÉTÉ" to "été" can now lose that anchor and fail the 0.90 anchor-retention gate even though the content was preserved. The emoji corruption being fixed is in the ASCII-stripping heading/marker paths, so the anchor extractor should either keep locale-aware normalization or explicitly strip/normalize its emitted anchors before sorting.
Useful? React with 👍 / 👎.
1 participant
Task #9 of the workflow-upgrade run (plan @ e6db065). Fixes the "report generation failed" WARNING seen on a real bounce run today.
Root cause — not the out-of-repo input path the bug report guessed (disproved with an ASCII fixture): gawk 5.0's locale-aware
gsub(/[^a-z0-9 ]/,...)under a UTF-8 locale strips 3 of 4 bytes of an emoji (the 🎯 in the input doc's heading), leaving an orphan continuation byte that makes locale-validatingsorthard-crash;set -euo pipefailexits the scorer (rc=2), sobounce-scores.jsonnever appears andreport-bounce.shdies — surfaced only as the auto-hook WARNING atco-evolve-bouncer.sh:684.What changed
evals/score-bounce.sh(~:122-201):LC_ALL=C awkon the three ASCII-fingerprint extractors (extract_headings/extract_anchors/extract_markers) — byte semantics are correct here since every multi-byte char is stripped either way; comment documents the constraint.tests/report-bounce-encoding-simulation.sh(new, 3 scenarios): emoji-heading fixture fails pre-fix with the exact real error signature, passes post-fix; ASCII out-of-repo fixture proves the path was never the trigger.Verification — new sim 3/3 (proven failing pre-fix);
tests/run-all.sh --quick24/24; slow suites skipped with grep-verified zero code-path overlap; replay against the actual failing run dir now completes and emits HUMAN-REPORT.md.Follow-up noted, not done: the auto-hook suppresses report-bounce stderr, which hid this crash's signature — orthogonal diagnosability gap.
🤖 Generated with Claude Code