fix(prepare): use raw byte length for BPB token_bytes — avoids UTF-8 replacement inflation

[warren618]

Summary

Fixes #384

The BPB token byte count is computed by decoding each token via tiktoken and re-encoding to UTF-8. For tokens whose raw bytes are not valid standalone UTF-8, tiktoken replaces them with U+FFFD (3 bytes), inflating the byte count and producing artificially lower BPB scores.

Fix

Use mergeable_ranks directly to get the true raw byte length, bypassing the decode/re-encode roundtrip:

rank_to_bytes = {rank: raw for raw, rank in mergeable_ranks.items()}
for token_id in range(enc.n_vocab):
    if token_id in rank_to_bytes:
        token_bytes_list.append(len(rank_to_bytes[token_id]))
    else:
        token_bytes_list.append(0)

Why this matters

BPB is the core metric driving the entire autoresearch loop. An inflated byte count denominator means experiments appear to perform better than they actually do.