Llama-3.2-3B-Instruct (w8a8_g128 conversion) stops generation after 2-13 tokens regardless of prompt or num_predict — resolved: specific to one uploader's conversion, not systemic

[Hi5808]

Update before filing

This turned out not to be an rkllm-toolkit or rkllama bug. Did a from-scratch rkllm-toolkit conversion of unsloth/Llama-3.2-3B-Instruct with identical settings (w8a8_g128, optimization_level=1, target_platform=rk3588, num_npu_core=3, max_context=4096) and it generates full-length, coherent responses with no truncation — a 199-token response to a 200-token budget, vs. 2-13 tokens on the original file. Same base model, same quantization scheme, clean result. So the bug is specific to jamescallander/Llama-3.2-3B-Instruct_w8a8_g128_rk3588's particular .rkllm upload (bad calibration run, corrupted file, or a conversion-time misconfiguration on their end) — not a general Llama-3.2 tokenizer/EOS issue, and not an rkllm-toolkit/rkllama bug.

Given this, consider not filing as-is. This isn't really an actionable bug report against either project anymore. Options: skip filing this one entirely, or file a short informational note (not a bug report) flagging that this specific HuggingFace upload appears broken, so other users searching for the same symptom find an answer. The original report below is kept for reference/context.

Original summary (kept for reference — see resolution above)

Every generation request against jamescallander/Llama-3.2-3B-Instruct_w8a8_g128_rk3588 stops almost immediately — 2 to 13 tokens generated, done_reason: "stop" — regardless of prompt content and even with options.num_predict explicitly set to 100-256. The same-author, same-quantization-scheme conversion of TinyLlama-1.1B (jamescallander/TinyLlama-1.1B-Chat-v1.0_w8a8_g128_rk3588) shows no such issue and reliably generates full-length, coherent responses. This is reproducible: 6/6 trials in our benchmark run were affected, across three different prompts.

I'm filing this against rkllama since that's the runtime path we exercised it through, but I want to flag up front that the root cause could plausibly sit anywhere in the stack: the specific HF conversion, rkllm-toolkit's handling of the Llama 3.2 tokenizer/EOS tokens during quantization, or rkllama's chat-template/stop-token handling. Filing here as the most visible place people would hit this via the API, happy to have it redirected if it's actually upstream in rknn-llm or a conversion-specific problem.

Environment

• rkllama version: 0.0.75
• rkllm-runtime: 1.3.0
• Model: jamescallander/Llama-3.2-3B-Instruct_w8a8_g128_rk3588 (converted with rkllm-toolkit 1.2.1, max_context_limit: 4096)
• Tokenizer/HF path used for chat templating: unsloth/Llama-3.2-3B-Instruct (used as a stand-in since meta-llama/Llama-3.2-3B-Instruct is gated)
• Board: Radxa ROCK 5B+ (RK3588), Armbian 26.8.1

Steps to reproduce

curl -s -X POST http://localhost:8080/api/chat -d '{
  "model": "Llama-3.2-3B-Instruct_w8a8_g128_rk3588",
  "messages": [{"role":"user","content":"List five random fruits and describe each one in one sentence, focusing on taste and texture."}],
  "stream": false,
  "options": {"num_predict": 200}
}'

Actual

{
  "eval_count": 13,
  "done_reason": "stop",
  "message": {
    "content": "The concept of artificial intelligence (AI) has been around for centuries,",
    "role": "assistant"
  }
}

(Note: that content doesn't even match the fruit-listing prompt above — this was from a different trial with the same truncation pattern; included to show the truncation is consistent even when responses otherwise vary.) Across 6 trials with 3 different prompts, eval_count ranged 2-13 tokens every time, with done_reason: "stop" (not "length"), meaning the model itself is emitting an early stop/EOS token, not hitting num_predict.

Expected

Full-length responses proportional to num_predict, similar to what the same-author TinyLlama-1.1B conversion (w8a8_g128, same rkllm-toolkit version) reliably produces.

What I've ruled out

• Not a num_predict/Modelfile config issue — tested at 100 and 256, no difference.
• Not a client/timeout issue — requests complete quickly and return valid JSON, just with almost no generated content.
• Not specific to raw-prompt vs chat-templated calls — reproduces identically via both /api/generate and /api/chat.
• Not a gated-tokenizer artifact — same behavior whether HUGGINGFACE_PATH points at the (gated) meta-llama/Llama-3.2-3B-Instruct repo (fails outright with a 401) or the ungated unsloth/Llama-3.2-3B-Instruct mirror (loads fine, same truncation).

Suggested next step for whoever picks this up

A from-scratch rkllm-toolkit conversion of Llama-3.2-3B-Instruct (rather than using this specific pre-converted HF repo) would help isolate whether this is a conversion-specific quantization/calibration artifact for this one upload, or a systemic issue with how rkllm-toolkit or rkllama handles Llama 3.2's tokenizer/EOS tokens.

Additional context

Full writeup with the underlying benchmark data (this is called out as Finding 3): https://claude.ai/code/artifact/09b27c30-c837-4204-8941-1d107ee75c9d

[danielferr85]

Use models converted with version 1.3.0 rkllm-toolkit.