While comparing a g5.8x to an r8gd.8x,
I noticed a 50x difference in latency between the CAGRA_HNSW and HNSW query latencies on a 10M dataset when a 1M subset only saw about a 15% difference. I think the issue is explained by the large Java Heap that was required for indexing could evict the cached data seen during indexing, which doesn't happen on CPU since the CPU System RAM is twice as large. Will test adding in prewarmup and cold-start options to see if that resolves the observed discrepancies (where prewarmup refers to clearing cache and then fully/partially caching the index, and where cold-start refers to simply clearing the cache beforehand).
Note: this fix I have in mind should also resolve weird efSearch results which appear to benefit from cache-hits too much.
NOTE: I'm testing a possible fix for this.
While comparing a g5.8x to an r8gd.8x,
I noticed a 50x difference in latency between the CAGRA_HNSW and HNSW query latencies on a 10M dataset when a 1M subset only saw about a 15% difference. I think the issue is explained by the large Java Heap that was required for indexing could evict the cached data seen during indexing, which doesn't happen on CPU since the CPU System RAM is twice as large. Will test adding in prewarmup and cold-start options to see if that resolves the observed discrepancies (where prewarmup refers to clearing cache and then fully/partially caching the index, and where cold-start refers to simply clearing the cache beforehand).
Note: this fix I have in mind should also resolve weird efSearch results which appear to benefit from cache-hits too much.
NOTE: I'm testing a possible fix for this.