- Support disaggregated prefill and decoding.
- Support KV Cache aware routing.
- Support KV Cache Pool.
- Support instance incarnation tracking and lease-lost failover.
- Support tool call and reasoning parser in PD disagg mode.
- Support chat_template_kwargs in chat completions.
- Support service to respond cancel status to instance upon client disconnection.
- Optimize kv cache hash algorithm and improve performance.
- Replace all http with rpc for interaction with xLLM instance.
- Support multi xllm_service sending request to single xllm instance.
- Add bvar to monitor xllm service metrics.
- Fix incorrect behavior when prefill instance shutdown and first token not form str.
- Fix hang issue caused by unreleased requests after instance shutdown.
- Set correct parameters for client output.