Roadmap question: multi-session / concurrent multi-user inference support

[AndreaSuriano96]

Hi,

thanks for the great work on ds4.

I would like to ask whether multi-user / multi-agent serving is part of the roadmap.

As far as I understand, ds4-server currently supports multiple clients, but generation is still serialized through a single inference worker. This makes sense for a single interactive session, but it limits use cases where multiple coding agents are connected at the same time.

The feature I would be interested in is not necessarily full continuous batching immediately, but at least:

• multiple independent sessions
• stable session routing, for example through an X-Session-Id header
• separate KV/cache state per session
• warm session pool
• LRU eviction for inactive sessions
• priority/queue scheduling between clients

In the longer term, continuous batching would obviously be ideal, but even a multi-session pool would already make ds4 much more practical for local multi-agent workflows on systems like DGX Spark.

Is this something you plan to implement, or would contributions in this direction be welcome?

Thanks!

[antirez]

The hardware this project targets, Metal, small CUDA systems like the GB10, are not effective at doing that, and also the MoE nature of DS4F needs serious batching to leverage grouping experts per layer in order to reduce significantly the memory bandwidth and making decoding more compute bound. But you said that it's more the API interface that you would like to see working, but isn't this already kinda working with the on disk KV-cache? The loading delay is not big, while taking even just a few KV cache sessions on RAM looks very costly. Consider that the OS page cache is already helping us. If the just saved kv-cache session gets reloaded, we are likely going to load from RAM. So my feeling is that there is a bit of a mismatch about the hardware and memory sizes we target and the idea of parallel sessions. However having optional RAM KV caches could be simple to support, and may worth it in the future. I'll think about it (I'm already doing it). First I want to try to stabilize what we have and support well the hardware, so that later we can extend this kind of things :) Thanks.