feat: client-side compression/dedup via VDO (issue #277)

[boddumanohar]

Summary

Implements the design in operator/docs/designs/design-issue-277-client-side-compression.md (PR #398) and its test plan in operator/docs/tests/test-plan-issue-277-client-side-compression.md, closing issue #277.

• New Pool.spec.storageClassParameters.clientCompression / clientDeduplication fields, independently switchable, mapped through to client_compression/client_deduplication StorageClass parameters.
• upsertStorageClass now composes a VDO-capable topology requirement (simplyblock.io/vdo-capable=true) with the existing DHCHAP topology gate when either client-side parameter is set.
• New csi-driver/pkg/util/vdo.go: CreateOrAttachVDO, ResolveClonedVDO, DeactivateVDO, RemoveVDO (with a dmsetup-based fallback for the orphaned-stack case documented in the design's spike log), GrowVDO, SetVDOFeatures. Each volume gets its own PV/VG/vdo-pool/LV stack via LVM (lvcreate --type vdo).
• nodeserver.go wiring: NodeStageVolume creates/attaches VDO between initiator.Connect and mount; NodeUnstageVolume deactivates it before disconnecting the raw device; restageVolume reattaches (never recreates) on reconnect; NodeExpandVolume grows the VDO stack before the filesystem resize; xfsStripeOptions is skipped once VDO is in play; buildAccessibleTopology surfaces the new node label.
• Node capability: the CSI node DaemonSet's postStart hook installs/loads kmod-kvdo+vdo via nsenter (added hostPID: true), writing a marker file; newNodeServer reads it in the background and patches the node's simplyblock.io/vdo-capable label via a new RBAC patch/update grant on nodes.
• csi-driver/deploy/image/Dockerfile now installs lvm2+vdo (amd64 only) and disables udev sync for LVM commands, both required to actually run LVM/VDO tooling from inside this container.

Verified end-to-end on the live test cluster

Created a real Pool with both clientCompression and clientDeduplication enabled, provisioned a PVC/Pod through the resulting StorageClass, and confirmed:

• StorageClass correctly carries client_compression/client_deduplication = "True" and an allowedTopologies requiring vdo-capable=true.
• The PVC/Pod is scheduled only onto the one node in the cluster whose kernel actually has a matching kmod-kvdo build (the other 3 nodes correctly advertise vdo-capable=false).
• The pod's mount is a genuine VDO device (/dev/mapper/vdo-<lvol-uuid>-...), with VDOCompression/VDODeduplication both enabled (lvs).
• Real compression+dedup savings: writing ~104MB across one original file and two exact duplicates of it yielded an 89% vdostats saving percent.
• Re-provisioning: deleting and recreating the pod (same PVC, same node) reattaches the same VDO device rather than recreating it — data survives with an exact checksum match.
• Expansion: growing the PVC 6Gi → 10Gi correctly grows the VDO pool and the filesystem online, with data intact throughout.
• Multiple instances across a reboot: two real PVCs with distinct, checksummed data on the same node, node rebooted — both VDO instances reattached cleanly via fresh NodeStageVolume calls (kvdo module usage count exactly 2, both VDOOperatingMode: normal, both checksums matched). Caveat: the two stage sequences happened to run sequentially rather than genuinely overlapping, so LVM's internal locking under truly concurrent vgchange/pvscan calls remains unexercised.
• Clone/snapshot resolution: both a direct PVC-to-PVC clone and a snapshot restore, each scheduled onto the same node as their still-live source (the specific co-location risk the design doc calls out) — ResolveClonedVDO correctly ran vgimportclone + lvrename, both mounted cleanly with data matching the source exactly, and all three volumes (source, clone, restore) coexisted with independent VG identities and no cross-contamination.
• Stale state after an unclean disconnect: forcibly severed a real VDO volume's NVMe-oF connection at the host level (nvme disconnect) while the node and pod both stayed up, simulating "storage side disconnects while the node stays up." Kernel logs showed buffered reads/writes silently appearing to succeed for ~19s before the real failure surfaced, at which point VDO correctly fenced itself into read-only mode (and ext4 independently aborted its journal) rather than corrupting anything — Kubernetes reported the pod as healthy the whole time regardless. Deleting the pod afterward is now confirmed to clean up fully automatically, with no orphaned state left behind and no manual intervention needed.
• XFS on top of VDO: provisioned a real VDO volume with fsType=xfs (every prior spike used ext4) — mkfs.xfs ran with no stripe-alignment flags (confirming the xfsStripeOptions skip fires correctly), mounted cleanly with the existing nouuid flag intact, compression/dedup stayed enabled, and reattach-on-recreate worked identically to ext4. No bugs found.
• Crash-consistency of async write policy: forced vdo_write_policy=async explicitly, wrote one file with fsync() and one without, then genuinely crashed the node via sysrq (immediate reboot, zero filesystem sync — confirmed via a new boot timestamp, not a graceful reboot that would have proven nothing). The fsync()'d file survived with an exact checksum match; the non-fsync()'d file was lost entirely — the correct POSIX outcome. Resolves the design doc's open safety question: async correctly honors flush/FUA durability end-to-end through NVMe-oF to the simplyblock backend.

This process found and fixed nine real, hands-on-only-discoverable bugs beyond the original design (each is its own commit on this branch):

1. buildAccessibleTopology only added the vdo-capable topology key when the label was already true — since CSINode's topology key set is captured once at plugin registration (seconds after pod start) while the label gets patched asynchronously afterward, the key was essentially never present at registration time, permanently breaking the topology gate until pod restart.
2. The spdkcsi container image had no lvm2 installed at all (pvcreate: not found).
3. lvm2 alone wasn't enough — lvcreate --type vdo shells out to vdoformat, which ships in the separate vdo package.
4. lvcreate --type vdo failed with "device not cleared" — no udev daemon runs inside this container, so device-mapper's default udev-sync handshake never completes. Fixed with DM_DISABLE_UDEV=1.
5. Most serious: NodeUnstageVolume was calling the destructive RemoveVDO (vgremove) on every routine unstage, not only when the volume was actually being deleted — an ordinary pod delete+recreate on the same node silently destroyed all VDO-backed data. Fixed by adding a non-destructive DeactivateVDO (vgchange -an only) for this path.
6. GrowVDO's lvextend -l100%FREE used the absolute (not additive) percentage form, which computed a target smaller than the pool's current size on every real device resize. Fixed with -l+100%FREE.
7. pvVGName trusted pvs's combined stdout+stderr output wholesale, so a duplicate-PV WARNING: line from a real clone test polluted both the identity comparison and the resulting log message. Harmless in that run (the dirty string still differed from the target either way), but fixed to parse just the actual field.
8. DeactivateVDO (the non-destructive replacement from bug updated the api create param to include cr objects #5) had no fallback at all for an unreachable backing device — vgchange -an failed identically to how vgremove failed in the original spike, on every one of 18 retries, until kubelet gave up and force-removed the pod anyway, leaving the orphaned dm-vdo stack permanently stuck. Fixed by adding the same dmsetup remove fallback RemoveVDO already had.
9. That fallback's device-name matching didn't account for device-mapper's dash-escaping (vdo-<uuid> → vdo--<uuid-with-double-dashes> in dmsetup ls output), so it matched nothing and the stack stayed orphaned even with the fallback wired in. Fixed to escape the VG name the same way device-mapper does before matching.

Deliberate scope decisions (see design doc for full rationale)

• ResolveClonedVDO's clone-collision detection is driven purely by the device's actual on-disk VG identity, not by threading VolumeContentSource through the volume context as the design doc originally proposed — simpler and correct either way, so no new plumbing was added.
• GrowVDO's signature grows to the pool's new physical capacity (-l100%FREE/+100%FREE, then matching logical size) rather than taking an explicit newSize parameter, matching the 100%FREE convention already used at creation time.
• VDO code lives in csi-driver/pkg/util/vdo.go per the design doc, not atlas-lib (raised as an idea mid-discussion, never confirmed).
• lvm2/vdo are installed in this branch-tagged Dockerfile, not the shared base_image/Dockerfile_base — rebuilding that shared, cross-branch tag for an in-progress feature felt like the wrong blast radius; worth promoting there once this lands.
• vdo has no aarch64 build in the configured repos, so it's installed amd64-only; client-side VDO is x86_64-only for now, matching every host used in this design's validation.

New finding, not yet fixed (separate, pre-existing gap)

The CSI driver's CreateVolume does not populate PersistentVolume.spec.nodeAffinity, so once a PVC is bound, a pod using it can be rescheduled to any node — not just the one the topology gate originally selected. This was harmless before (a raw NVMe-oF connection works identically from any node) but is now materially important, since VDO state is node-local. Reproduced live: deleting and recreating the pod (not the PVC) let it land on a non-vdo-capable node, where it correctly failed to mount. Worked around for this verification by pinning the test pod's nodeSelector; not fixed here since it's a pre-existing driver gap orthogonal to this feature (same category as #401). Filing a follow-up issue.

Explicitly out of scope for this PR

• SetVDOFeatures (live compression/dedup toggle) is implemented but not wired into any update path — v1 non-goal per the design doc.
• The pre-existing upsertStorageClass create-only bug (bug: upsertStorageClass is create-only — Pool StorageClassParameters edits on an existing Pool silently no-op #401) — orthogonal, tracked separately.
• Whether kubelet reliably re-invokes NodeUnstageVolume on the original node specifically after that node goes NotReady and later rejoins (as opposed to the connection just being severed while the node stays healthy, which is what was tested) remains unverified — a different code path.
• Genuinely concurrent (not just closely-timed) vgchange/pvscan calls actually racing at the LVM level remain unexercised — the multi-instance reboot test's two stage sequences happened to run sequentially rather than overlapping.

All items originally flagged as open in the test plan (XFS-on-VDO, crash-consistency of async, clone/snapshot resolution, multi-instance-across-a-reboot, stale-state-after-unclean-disconnect) are now verified — see above.

• ensureDeviceConnected (block-volume reconnect) is untouched — every VDO example in the design is filesystem-mode; block-mode + VDO was never designed or tested.
• Minor: NodeExpandVolume/GrowVDO isn't fully idempotent against a redundant re-invocation after the volume is already at its target size — logs a scary-looking but harmless error on kubelet's post-success reconciliation retry. Worth a follow-up polish pass.

Why draft

Unit tests from the test plan are not written yet. Draft status reflects that, not any doubt about the functional behavior above — that part is now hands-on verified against a real cluster, real NVMe-oF-backed lvols, and real compression/dedup savings.

Test plan

• go build ./... clean for both operator and csi-driver modules
• go test ./... — pkg/spdk, pkg/util pass; internal/controller's envtest-based suite requires a local kubebuilder binary not present in this environment (pre-existing, unrelated); e2e package requires a live e2e cluster fixture (pre-existing, unrelated)
• Manual end-to-end verification on the live test cluster (Pool → StorageClass → PVC/Pod → VDO device/compression/dedup confirmed → reattach-on-recreate confirmed → expand confirmed → multi-instance reboot confirmed → clone/snapshot resolution confirmed → stale-state cleanup after unclean disconnect confirmed → XFS-on-VDO confirmed → async write-policy crash-consistency confirmed via a genuine node crash)
• Unit tests for vdo.go and the new wiring (test plan Sections 1-7)

🤖 Generated with Claude Code