feat(B9.6e): delete dead hxtok.c BPE C library (.c 228→227) - #1820
Merged
Merged
Conversation
self/native/hxtok.c (750L) + hxtok.h (49L) — a standalone Qwen2.5 BPE tokenizer C lib — is a dead shim: zero build scripts reference it (tool/*, *.json, *.sh), zero .so/.dylib artifacts exist, and zero code FFI-calls HxTok/hxtok_* anywhere in the repo (the only repo-wide match is an archived-roadmap text literal in compiler/roadmaps_archive, not a code reference). All 8 listed consumers (self/ml/qwen_bpe, tokenizer, tokenizer_bpe, tokenizer_test, tokenizer_trainer, self/test_tokenizer*, stdlib/flame/flame_bpe_corpus_*) use the pre-existing pure-hexa BPE path (qwen_bpe.hexa / tokenizer_bpe.hexa), never the C lib. After deletion all 8 consumers parse cleanly, proving the C lib carried no live behavior — RUNEQ is moot (zero live callers). Not in runtime.c (standalone). This is the v565_grad_analysis.c (#1818) / crypto_blowfish (#1816) dead-file git-rm pattern, NOT a from-scratch port. RUNTIME.flip B9.5a flipped [⛔DUP]→[x]; audit doc reclassified. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
# Conflicts: # RUNTIME.flip.md
3 tasks done
dancinlife
added a commit
that referenced
this pull request
May 27, 2026
#1821) self/native/hxvocoder.c (455L) — a standalone HEXA-SPEAK neural-vocoder C lib (additive synth sin/exp/clip hot-loop, libm-only) — is a dead shim. Re-verified from scratch (the docs/runtime_native_c_layer_audit.md "layer ① port candidate, neural_vocoder.hexa reference exists" verdict was wrong on two counts): all 8 exported symbols (hxvocoder_decode_nv / _decode_wave / _linear_proj / _synth_additive / _tanh_vec / _vec_zeros / _version / _write_wav) have ZERO references in any .hexa/.c code file (only repo-wide matches are archived text in roadmaps_archive). The sole build reference is a file_exists()-guarded optional link in tool/build_native.hexa that auto-skips once the file is gone — that dead block is removed too. No .h/.so/.dylib/.o artifact, no build_toolchain .json entry, no dlopen. Not #include'd by runtime.c (standalone). The audit's claimed neural_vocoder.hexa does not exist in-tree; the "1,922x vocoder" is the pure-hexa Griffin-Lim path in self/ml/speech_audio.hexa. clang -fsyntax-only self/runtime.c EXIT=0 (untouched), zero remaining hxvocoder_* symbol refs in code post-deletion. This is the hxtok (#1820) / v565 (#1818) / blowfish (#1816) dead-file git-rm pattern, NOT a port. B9.6f re-verification ledger (the audit's 4 "primary suspects"): only hxvocoder is DEAD. hxflash_linux / hxlayer_linux / hxvdsp_linux are LIVE — each has real extern-fn FFI consumers + a dedicated build script (self/ml/hxflash.hexa+gpu_train, self/ml/hxlayer.hexa+bench/test, bench/hxblas_linux.hexa @link("hxvdsp")) — they need a multi-session port, NOT a clean delete. All other standalone native .c (gpu_codegen _stub, hxffi_slot, hxblas, hxccl, hxlmhead, hxqwen14b/32b, lora_cuda _host, hexa_cc) re-scanned = all LIVE with callers/build-refs (PRESERVE). RUNTIME.flip B9.6f [x]; audit doc reconciled (B9.6d table corrected). Co-authored-by: dancinlife <search5599@proton.me> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
self/native/hxtok.c(750L) +hxtok.h(49L) — a standalone Qwen2.5 BPE tokenizer C library — dropping the.cfloor 228 → 227 toward the.hexa-only north-star (RUNTIME.flip B9.6e).v565_grad_analysis.cfeat(B9.6d): delete dead v565_grad_analysis.c harness (.c 229→228) #1818 /crypto_blowfish.cfeat(B9.6c): delete crypto_blowfish.c — rewire bcrypt_pbkdf to hexa-native blowfish (.c 230→229) #1816), NOT a from-scratch port.Why it's a clean dead-file removal (investigated read-only first)
tool/*,*.json, or*.shbuilds or linkshxtok/libhxtok; no.so/.dylibartifacts exist.grepforHxTok/hxtok_load/hxtok_encode/hxtok_free/hxtok_special_id/hxtok_vocab_size/hxtok_merges_count/hxtok_version_v1finds no code reference outside the lib itself. The single match (compiler/roadmaps_archive/embedded.gen.hexa) is an archived-roadmap text literal (libhxtok.soinside a frozen JSON string), not a code reference.self/ml/qwen_bpe.hexa("Qwen2.5 BPE tokenizer (pure hexa)") +self/ml/tokenizer_bpe.hexaalready implement byte-level BPE reading the same HuggingFacetokenizer.json. All 8 listed consumers use this hexa path; noneextern/FFI the C lib.#included byruntime.c(standalone) → no wipe-prone runtime surface touched.RUNEQ note (honest)
A live RUNEQ between
hxtok.cand the hexa path is moot: the C lib has zero live callers, so it carried no behavior to break. The equivalence evidence is structural — after deletion all 8 consumers parse cleanly (hexa parseOK on qwen_bpe / tokenizer / tokenizer_bpe / tokenizer_test / tokenizer_trainer / self/test_tokenizer* / flame_bpe_corpus_lib / flame_bpe_corpus_test), proving they never depended on the deleted file.Docs reconciled
RUNTIME.flip.mdB9.5a flipped[⛔DUP]→[x], new[x] B9.6eentry added.docs/runtime_native_c_layer_audit.md—hxtok.crow + layer-① list + priority list reclassified as DELETED.Test plan
.ccount:find . -name '*.c'= 227 (was 228)hexa parsecleanly post-deletionHxTok/hxtok_*code callers🤖 Generated with Claude Code