Skip to content

Fix compact context length parsing in preprocess - #10

Open
mySebbe wants to merge 1 commit into
Infini-AI-Lab:mainfrom
mySebbe:fix-preprocess-compact-lengths
Open

Fix compact context length parsing in preprocess#10
mySebbe wants to merge 1 commit into
Infini-AI-Lab:mainfrom
mySebbe:fix-preprocess-compact-lengths

Conversation

@mySebbe

@mySebbe mySebbe commented May 14, 2026

Copy link
Copy Markdown

Summary

  • parse compact result lengths such as 8k/8K into numeric token counts in preprocess.py
  • keep numeric lengths such as 0, 8000, and 16000 working as before
  • move preprocessing into small functions so the parser can be unit tested without running the whole script on import
  • add regression coverage for the compact-length case reported in length parsing error in preprocess.py #8

Verification

  • python -m unittest discover -s tests -v
  • python -m compileall -q gsm-infinite tests
  • reproduced the original failure with length: 8k, then verified python gsm-infinite/preprocess.py processes the same result file successfully

Fixes #8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

length parsing error in preprocess.py

1 participant