fix: don't flag codex/hermes runs as empty_run on unparseable token stats#13
Open
Wanli-Lee wants to merge 1 commit into
Open
fix: don't flag codex/hermes runs as empty_run on unparseable token stats#13Wanli-Lee wants to merge 1 commit into
Wanli-Lee wants to merge 1 commit into
Conversation
…tats _classify_agent_status called a run "empty_run" whenever aggregate_chat_jsonl reported n_calls=0 / output=0. But aggregate_chat_jsonl only understands openclaw's message.usage records — on codex/hermes (different chat.jsonl schema) it returns zero counts even for a full multi-MB transcript. This mislabeled real, high-scoring codex runs (e.g. a task that scored 0.984) as empty_run, polluting the agent_status / error fields with false positives. Guard the empty_run verdict with the chat.jsonl size: a transcript at or above NONEMPTY_CHAT_BYTES (4 KB) proves the agent ran, so we never call it empty on the strength of an uncountable token stat alone. Genuine empty/failed runs leave a tiny or absent chat.jsonl and are still caught. Only the agent_status / error labels were affected — scores were never touched. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_classify_agent_status(added in #10) labeled a run empty_run wheneveraggregate_chat_jsonlreportedn_calls=0/output=0. But that aggregator only understands openclaw'smessage.usagerecords — on codex/hermes (different chat.jsonl schema) it returns zero counts even for a full multi-MB transcript. This mislabeled real, high-scoring codex runs as empty_run.Concrete false positive: a codex task that scored 0.984 had
chat.jsonlof 4.9 MB butn_calls=0, so it was taggedagent_empty_run: clean exit but no model output. ~80/92 codex tasks in a live run were affected.Fix
Guard the empty_run verdict with the
chat.jsonlfile size: a transcript ≥NONEMPTY_CHAT_BYTES(4 KB) proves the agent ran, so we never call it empty on the strength of an uncountable token stat alone. Genuine empty/failed runs leave a tiny or absentchat.jsonland are still caught.Scores were never affected — only the
agent_status/errorlabels. This just stops the labels from being noise on codex/hermes.Test plan
pytest tests/— 73 passed (addedtest_classify_not_empty_when_chat_is_largeregression; existing empty_run tests now pass explicitchat_bytes)ok, notempty_run🤖 Generated with Claude Code