Follow-up from #169, filed rather than blocking that merge.
#169 makes the Codex runner fall back to the saved session transcript from ~/.codex/sessions when codex exec --json does not echo the full JSONL to stdout. That is a clear improvement over recording nothing, which is what happened before. The open question is whether the two sources are interchangeable downstream.
They are both JSONL but appear to carry different event schemas:
- the session file uses
turn_context with a payload wrapper, which is what extractObservedModelFromCodexSession reads (payload.model, payload.collaboration_mode.settings.model)
packages/agent-eval/src/lib/o11y/parsers/codex.ts switches on item.started / item.completed, plus response.failed and output_text.delta
If that reading is right, a fallback transcript will not crash, there is a default case, but it may parse to near-zero events. The concern is that transcript-derived numbers would then read as 0 rather than absent, which is worse than a null transcript when results are aggregated, because zero looks like data.
I could not settle this without a real Codex run, so it needs either confirmation that the schemas line up, a normalizer on the fallback path, or scoping the fallback so it populates the raw on-disk transcript without feeding the parsed path.
cc @huang-julien
Follow-up from #169, filed rather than blocking that merge.
#169 makes the Codex runner fall back to the saved session transcript from
~/.codex/sessionswhencodex exec --jsondoes not echo the full JSONL to stdout. That is a clear improvement over recording nothing, which is what happened before. The open question is whether the two sources are interchangeable downstream.They are both JSONL but appear to carry different event schemas:
turn_contextwith apayloadwrapper, which is whatextractObservedModelFromCodexSessionreads (payload.model,payload.collaboration_mode.settings.model)packages/agent-eval/src/lib/o11y/parsers/codex.tsswitches onitem.started/item.completed, plusresponse.failedandoutput_text.deltaIf that reading is right, a fallback transcript will not crash, there is a
defaultcase, but it may parse to near-zero events. The concern is that transcript-derived numbers would then read as0rather than absent, which is worse than a null transcript when results are aggregated, because zero looks like data.I could not settle this without a real Codex run, so it needs either confirmation that the schemas line up, a normalizer on the fallback path, or scoping the fallback so it populates the raw on-disk transcript without feeding the parsed path.
cc @huang-julien