[WIP][feat] prompt-native generative recommendation

[WhiteSwan1]

WIP — opened for design review, not for merge. Multi-rank is untested and two
items from the design remain unbuilt (see Not done below).

Replaces the SID-feature-based generative stack with a prompt-native one: a new
prompt_config owns the template, slots, SID space and tokenizer, and
model_config keeps only what belongs to the LM.

Why

The per-level SID offset used to be applied twice — once in the SID tool's
sid_offset_codes, once again in SidFeature._parse — with no hash covering the
pair. The collision tool now emits an offset_codebook column, so the offset
arrives already applied and that second implementation can go.

What remains inside tzrec is a uniform + base_vocab, which is not
position-dependent and therefore needs no bespoke feature type. SidFeature is
deleted; an INLINE slot is an ordinary sequence_raw_feature and a PROJECTED one
is a sequence_id_feature with an injected num_buckets.

The offset form is also self-validating: a raw code 100 is legal under codebook
4096 and 8192 alike, so a tool/config mismatch passes a range check, whereas
offset codes carry cumsum(codebook[:l]) inside the value and throw every level

= 1 out of band.

What is here

area
tzrec/protos/prompt.proto	PromptConfig, PromptSlot, PromptProjection, SidSpace
tzrec/prompt/compile.py	compiles the config into a resolved SidSpace, a PromptPlan, a ModulePlan and an extended tokenizer
tzrec/prompt/assembler.py	builds the packed varlen token stream in the dataloader worker
tzrec/prompt/persist.py	writes the contract beside the weights, checks it on restore
tzrec/models/prompt_generative_qwen.py	the model: inputs_embeds forward, index_copy scatter, band-constrained decode
tzrec/modules/prompt_projection.py	slot width -> LM hidden size
docs/source/models/prompt_generative_qwen.md	operator manual (Chinese)

Removed: sid_feature.py, generative_model.py, generative_qwen.py,
generative_model.proto and their tests. Nothing is kept for compatibility.

Two properties worth reviewing specifically:

• The compiler resolves no dimension. No artifact stores in_dim/out_dim;
  the model resolves both at __init__ from group_total_dim and the backbone
  config, so one compiled plan is valid across model sizes.
• No shape is discovered on the device. max_seqlen is computed by the
  collator on the host and carried in additional_infos; the model never calls
  lengths.max().

Test Plan

Unit: 55 tests across compile, assemble, projection, persistence and the model.
pre-commit run -a and python scripts/pyre_check.py both clean.

End to end, on a two-layer Qwen saved locally (no download), both an INLINE-only
prompt and one with an added PROJECTED slot:

step	result
tzrec.train_eval	3 steps, ce_loss 4.79 against ln(128) = 4.85 — near-random from a fresh init, as expected
eval	ce_loss computed and written to train_eval_result_v2.txt
checkpoint	model.ckpt-N/prompt/ carries sid_space.json, prompt_plan.json, prompt_hashes.json, tokenizer/
--continue_train	restores and continues
hash guard	changing the codebook to 8,8,8 is refused at restore
tzrec.predict	64 rows in, 64 out; generated_sids of shape (num_return, num_levels); every code inside [0, codebook), so the bands held and detokenize inverted both shifts
tzrec.export (HF)	loads with AutoModelForCausalLM.from_pretrained; export dir carries the prompt contract

Eight defects were found by running the pipeline rather than by unit tests —
four missed _create_model call sites, a (total, value_dim) shape the
assembler mis-read, the absent loss/metric hooks, an FX Proxy in decode, and
the un-derived feature groups a projected slot needs. All are fixed and covered.

Not done

• Multi-rank is untested. Everything above is --nproc-per-node=1, so
  sharding, DMP and the planner have not been exercised. The projected config is
  the one with a sparse table to shard.
• The scripted serving front end and cache_ids (§11 of the design). Without
  cache_ids, a stock radix prefix cache keyed on token ids will reuse KV across
  users at a sentinel position — a correctness bug, so it must land before any
  projected slot is served.
• TorchScript export is refused with a message pointing at export_format: HF;
  the model's input is a stream the dataloader assembles, which an export-time
  dummy batch cannot supply.

Open design question

design_v2.md §8.1 has an "INLINE text" branch, but TokenizeFeature extends
IdFeature and is therefore always sparse, hence always PROJECTED. No existing
type can be an INLINE text slot. I implemented INLINE as unconditionally meaning
SID. Either the branch goes, or TokenizeFeature gains a no-embedding mode.