On /v1/chat/completions, ollama_chat_to_openai_v1_chat_completion (format_utils.py:480, and the streaming path :672) does eval_duration_ns / 1e9 if eval_duration_ns > 0 else 0. When the worker returns eval_duration: None (e.g. very short/aborted generations or prompt-cache hits), .get("eval_duration", 0) returns None (key present, value None), so None > 0 raises. The 500 then cascades: the worker's child_conn.send(WORKER_TASK_ERROR) (worker.py:367) hits BrokenPipeError, the worker process dies, and subsequent requests fail with KeyError: '' (model unloaded).
Possible fix:
coerce to 0 with ... or 0 on all four *_duration reads in both the non-stream and streaming blocks; optionally wrap the worker's child_conn.send in try/except (BrokenPipeError, OSError).
tested on rkllama-runtime 1.2.3, RK3588, model qwen3-8b-16k
On /v1/chat/completions, ollama_chat_to_openai_v1_chat_completion (format_utils.py:480, and the streaming path :672) does eval_duration_ns / 1e9 if eval_duration_ns > 0 else 0. When the worker returns eval_duration: None (e.g. very short/aborted generations or prompt-cache hits), .get("eval_duration", 0) returns None (key present, value None), so None > 0 raises. The 500 then cascades: the worker's child_conn.send(WORKER_TASK_ERROR) (worker.py:367) hits BrokenPipeError, the worker process dies, and subsequent requests fail with KeyError: '' (model unloaded).
Possible fix:
coerce to 0 with ... or 0 on all four *_duration reads in both the non-stream and streaming blocks; optionally wrap the worker's child_conn.send in try/except (BrokenPipeError, OSError).
tested on rkllama-runtime 1.2.3, RK3588, model qwen3-8b-16k