Cluster Hot Slots Heatmap — Technical Design Doc - Part 1

[nassery318]

Addressing: #470
Add cluster slot heatmap visualization

Overview

This document describes the slot-level heatmap now present in the Hot Keys tab of the Activity view, available on clusters that already run the hot-slots collection mode. It adds a per-slot view of where the cluster's hot keys are concentrated — how many of them each slot holds and which node reported it — with a drill-down from any slot to the keys inside it, so a user can go from "these keys are hot" to "this slot, on this node, holds most of them" without leaving the screen.

Motivation

Hot keys were presented as one flat, cluster-wide ranked list. The list identifies what is hot but not where the heat is concentrated, and the slot is the unit an operator can actually act on — resharding, slot migration, and hash-tag redesign all operate on slots. This made two common diagnostic tasks slow:

• Hotspot localization — the node heatmap narrows a problem to a node, but a node owns up to 16,384 slots, so "node 3 is hot" is not yet an actionable finding.
• Hash-tag misuse — spotting a single tag that has forced a large working set into one slot, which no amount of resharding will relieve.

The slot each key came from was already known during collection and discarded before it reached the UI. Recovering it is what this change does.

Goal

The goal of the hot slots heatmap is to:

• Show where the cluster's top-ranked hot keys land across slots as a single grid, so concentration is visible at a glance rather than inferred row by row.
• Let a user select a slot and see the hot keys it holds, then click through to a key in the key browser.
• Add no additional commands against Valkey, deriving the new data from work already performed during collection.
• Reuse the visual language of the existing node heatmap so the two views read as one system.

Current State Before This Change

The Hot Keys tab collects data two ways. The default samples MONITOR output. The second, used when a node runs an LFU eviction policy with cluster-slot-stats-enabled, asks Valkey which slots are consuming CPU (CLUSTER SLOT-STATS ORDERBY CPU-USEC), scans the keys within each returned slot, and ranks them by access frequency.

The second path is inherently slot-scoped, but the slot each key came from was dropped before the response left the collector. Every key was folded into one cluster-wide top-N list. A node heatmap sat alongside the list and answered "which node," leaving nothing between the node and the individual key.

Implemented Design

The Slot Heatmap is a second tab inside the existing node heatmap modal, reached from the Hot Keys toolbar button now labelled simply Heatmap. The tab appears only in cluster mode with hot-slots collection active and at least one returned key carrying a slot, since CLUSTER SLOT-STATS has no standalone equivalent.

• Collection. Keys retain their slot association through the scan and frequency phases. The top-N heap carries the slot alongside the key and its frequency, so the collector returns [key, freq, slotId] instead of [key, freq]. The enrichment step passes the slot through untouched while adding size and TTL.
• Response shape. One channel, unchanged in cardinality: the ranked key list, truncated to a cluster-wide top-N, with a slot stamped on each entry. No separate per-slot summary is returned.
• Aggregation. The server merges duplicate keys across nodes as before, stamping the reporting node onto each record and keeping the first slot it sees for a given key.
• Presentation. Slots are derived in the browser by grouping the returned key list on its slot field. Each group is one tile. The tab renders summary chips, a filterable legend, a tile grid, and a drill-down panel for the selected slot.

Slot Data Used

Metric used to classify slots

Metric	What it is	Why we use it
Hot key count	Keys from the cluster-wide top-N that fall in this slot.	The information the grouping adds, and the basis for tile intensity.

Fields shown as reported

Field	What it is	Why we use it
Slot ID	The slot number, 0–16383.	The tile label, and the identifier an operator carries into a reshard or migration command.
Reporting node	The node whose collector returned the keys in this slot.	Connects the slot back to the node heatmap and to physical remediation.
Per-key size, TTL	Already surfaced in the hot keys table.	Reused unchanged in the drill-down so the two views agree.

cpu-usec, keys scanned, and summed frequency are not carried to the UI. Summed frequency is computed in the browser but used only to break ties when ordering tiles.

UI Component Breakdown

Summary chips

A row above the grid carrying the totals: hot slots in the sample, hot keys across them, and nodes represented. Where one or more nodes failed to report, a fourth item states how many, so a shrunken total is never presented as complete.

Slot tile and drill-down

Each tile shows its slot ID and hot key count, coloured by concentration and sized uniformly. Tiles are ordered by hot key count, then by summed frequency. Hovering reveals the slot ID, its share of the top-N, and the reporting node. Selecting a tile opens that slot's key list beside the grid, under a header repeating the same share, with size and TTL per key and the owning node in the footer. Clicking a key closes the modal and selects that key in the key browser, completing the path from cluster-wide concentration down to a single key. A line beneath the grid states that it shows the slots holding the top-N hot keys, not every slot in the cluster.

Slot Intensity Classification

Each slot is shaded across five intensity bands. The grid is the primary scanning aid — with dozens of slots in the sample, it must let a user find the few that matter without reading every tile. Intensity is driven by hot key count, normalized against the range present in the returned set, and the legend doubles as a multi-select filter: selecting one or more bands dims every slot outside them. The colour ramp, the band snapping, and the legend are shared with the node heatmap.

Band	Working definition	Normalized hot key count
Palest	Fewest hot keys among the sampled slots.	Bottom of the range
Low / Mid / High	Progressively more concentrated; the mid bands carry the shape of the distribution.	Interpolated between
Deepest	Holds the most hot keys in the sample and is the first place to look.	Top of the range

• Why not CPU — CPU already determined which slots were scanned, so colouring by it would re-encode the selection criterion and make every tile look hot. It is not currently carried to the UI at all.
• Bands are relative, not absolute — slots are judged against each other within the current sample, so the shape of the distribution stays legible even when absolute counts are small. The cost is that intensity is not comparable across refreshes. Absolute thresholds are deferred to Future Work.
• Empty range — where every sampled slot holds the same number of hot keys, and where only one slot is present, every tile currently renders at the deepest band. This is a known deviation from the intended behaviour, which is to render a flat distribution at the palest band so it does not read as a cluster-wide emergency.

Edge Cases

• Slot counts are top-N counts, not slot totals: a tile counts only the keys that survived the cluster-wide cutoff, so it understates the slot. The UI states the count as "N of your top M hot keys" rather than implying a total.
• Slots absent from the grid: a slot whose keys all fell below the cutoff does not appear, even where its CPU is what caused it to be scanned. The grid shows where the top-N landed, not which slots are hot.