[QUESTION] Cache Policy Code

[SourishW]

Thanks for the great paper.

I had a question about the implementation. In the paper I found that ShadowKV uses a cache policy to retain chunks that remain the same from decode step to decode step.:

But I was unable to find this in the code. There's only a comment that says that position IDs can be used to determine the cache hit rate.

I wasn't able to find the mechanism where it only fetches the necessary chunks.

[preminstrel]

Hi @SourishW , it is done in CUDA code: https://github.com/ByteDance-Seed/ShadowKV/blob/main/kernels/map.cuh