Summary
This proposes an externalAccess configuration block on ValkeyCluster that lets clients outside Kubernetes connect to a sharded cluster. The operator exposes each shard through its own Service, optionally announces a stable per-shard hostname, and (when asked) directs clients to those external endpoints using Valkey's client-facing announce settings. Throughout, node-to-node traffic (gossip and replication) stays on internal pod IPs, so the operator's existing reconciliation, which correlates CLUSTER NODES addresses with pod IPs, is unaffected. The work is delivered as four small, stacked PRs. Each one is independently useful and fully backward compatible: a cluster that doesn't opt in renders exactly as it does today.
Motivation
Today a ValkeyCluster is reachable only from inside the Kubernetes cluster. Every node announces its pod IP, and the only Service is a headless ClusterIP. Cluster clients follow MOVED/ASK redirects, and in cluster mode those redirects carry the target node's announced address, which is a pod IP that isn't routable from outside. So even if you put a LoadBalancer in front of the pods, the first cross-shard redirect sends the client to an unreachable address and the connection fails.
This blocks a real and common deployment: applications running outside the Kubernetes cluster (a different cluster, a VM fleet, on-prem services) that need to use a Valkey cluster hosted in Kubernetes. Users have asked for it, and other Redis/Valkey operators support some form of it.
This need has been raised before: discussion #77 opened the idea (LoadBalancer, Ingress, NodePort, and proxy options, plus TLS termination), and design proposal #259 sketched per-shard Services, reading back allocated ports, publishing endpoints in status, and optional Gateway/TLSRoute SNI routing. This RFC follows that direction and fills in the details: how to keep the cluster bus internal so the reconciler is untouched, how the announce settings are rendered per node, the NodePort allocation and read-back mechanics, and the optional Gateway API path. It starts with TCPRoute rather than TLSRoute/SNI, and leaves TLSRoute as a follow-up.
The core difficulty is that Valkey uses one announced address for both client redirects and the inter-node cluster bus. Naively announcing an external address would push gossip and replication over that external path too, and it would break the operator, which assumes the announced address equals the pod IP. The key enabler is that Valkey 9.0 separates the node-to-node announce settings (cluster-announce-ip / -port / -bus-port) from the client-facing ones (cluster-announce-hostname, cluster-announce-client-port / -client-tls-port, and the cluster-wide cluster-preferred-endpoint-type). That separation is what makes this feasible without disturbing the internal data plane, and 9.0 is already the operator's documented baseline since it's required for shutdown-on-sigterm.
Detailed Design
The feature is opt-in through a new spec.externalAccess block. When it's absent, nothing changes. When enabled, the operator does four things, which map to the four implementation layers.
1. Node identity (human-readable names). With external access on, each node announces cluster-announce-human-nodename set to its ValkeyNode name (for example cluster-sample-1-2), so failure events reference a readable name instead of only the opaque node ID. This rides the same container-argument seam that already sets --cluster-announce-ip. It's a small standalone win and a natural place to introduce the API.
2. Per-shard Services. The cluster controller reconciles one Service per shard, selecting that shard's pods. Each Service exposes one port per node, and each port is pinned to a single pod by giving the server container a node-unique port name (vk-n<idx>) and targeting it by name. Kubernetes only adds a pod to a Service port's endpoints if the pod declares a container port with that name, so the port lands on exactly one pod. This is what lets a client reach a specific primary or replica, which cluster redirects need.
For NodePort (the default), the operator does not assign port numbers. It lets Kubernetes allocate them, which is the only way to avoid collisions across clusters in a namespace, then reads the allocated ports back and records them on the cluster status. For LoadBalancer, each shard has its own external address, so the frontend ports are simply 6379 + nodeIndex.
3. Per-shard hostnames. With a domain set, each shard announces cluster-announce-hostname = <hostnamePrefix>-<shardIndex>.<domain> (prefix defaults to shard). This is metadata only. It appears in CLUSTER SLOTS and is useful for TLS SNI, but it doesn't change where clients are sent yet. The cluster bus keeps using pod IPs.
4. Client endpoint selection. Each node announces its external client port (read back from the shard Service in step 2) via cluster-announce-client-port, or cluster-announce-client-tls-port when TLS is enabled. Setting preferredEndpointType: hostname then flips cluster-preferred-endpoint-type cluster-wide, so MOVED/ASK and CLUSTER SLOTS hand clients the shard hostname and external port. A client connecting to any shard now gets correct cross-shard redirects to a routable hostname:port.
5. Gateway API exposure (optional). As an alternative to NodePort/LoadBalancer Services, the operator can expose the cluster through the Gateway API. With externalAccess.gateway set, it creates one TCPRoute per node, attaching to a Gateway the user already runs and forwarding to the node's port on the per-shard Service. The Gateway is then the only external surface, so the backing Service defaults to ClusterIP. The announced client port is the Gateway listener port, basePort + shardIndex*(replicas+1) + nodeIndex, which is deterministic and needs no read-back. Gateway API is an optional dependency: the operator checks at startup whether the TCPRoute kind is served, and only then registers the watch and reconcile path. On a cluster without the Gateway API CRDs, it behaves exactly as it would without Gateway support, and never references a type the API server can't serve.
Components:
- API:
ExternalAccessSpec on ValkeyClusterSpec, mirrored onto ValkeyNodeSpec; a ShardEndpoint status type on ValkeyClusterStatus.
- Cluster controller: a
reconcileShardServices step (Services first, before nodes, so the allocated ports are available), endpoint read-back into status, per-node port resolution, and scale-in Service cleanup.
- Node resources: the announce arguments rendered onto the server container when external access is enabled.
- Config: the one cluster-wide setting (
cluster-preferred-endpoint-type) added to the shared base config so it rolls through the existing config-hash path.
Reconcile ordering and rollout. Services are reconciled before nodes, so a NodePort is normally allocated and read back on the same pass that builds the node. The announced values live in the pod template, so a change flows through the operator's existing one-at-a-time, proactive-failover roll. No new rollout machinery is needed. The cluster-wide endpoint-type setting rolls through the existing config hash.
User workflow. Set externalAccess.enabled: true with a serviceType, and the operator creates the Services and reports the external ports under status.externalEndpoints. Add a domain to announce hostnames, create DNS records pointing those hostnames at the Services, then set preferredEndpointType: hostname to direct clients to them.
API Changes
spec:
externalAccess:
# Turn on external access (per-shard Services + client-facing announce config).
enabled: true
# NodePort (default), LoadBalancer, or ClusterIP. Defaults to ClusterIP when a
# Gateway is configured.
serviceType: NodePort
# Cluster (default) or Local. Local preserves the client source IP.
externalTrafficPolicy: Cluster
# Applied to each per-shard Service (e.g. external-dns / cloud LB controller).
serviceAnnotations: {}
# Prefix for shard hostnames; set per-cluster when sharing a domain.
hostnamePrefix: shard
# DNS domain for shard hostnames. When set, each node announces
# "<hostnamePrefix>-<shardIndex>.<domain>".
domain: valkey.example.com
# ip (default) or hostname. hostname directs clients to the shard hostname.
# Requires domain.
preferredEndpointType: hostname
# Optional: expose the cluster through a Gateway API Gateway instead of (or in
# addition to) Service-level exposure. Requires the Gateway API CRDs.
gateway:
gatewayRef:
name: public-gateway
# First Gateway listener port; node (shard, nodeIndex) attaches to
# basePort + shardIndex*(replicas+1) + nodeIndex.
basePort: 30000
status:
# Populated when external access is enabled. NodePorts are indexed by node.
externalEndpoints:
- shardIndex: 0
nodePorts: [31000, 31001]CRD validation is expressed as CEL, since the repo has no validating webhook: hostnamePrefix must be a DNS label, and domain is required when preferredEndpointType is hostname. NodePort numbers are not user-settable; Kubernetes allocates them.
User Stories
- As an operator running apps outside the Kubernetes cluster, I want my external services to connect to a Valkey cluster hosted in Kubernetes, including correct cross-shard redirects, so I don't have to co-locate every client.
- As a platform operator, I want the external ports the operator ended up using surfaced on the resource status, so I can configure firewalls and DNS without inspecting individual Services.
- As a security-conscious operator, I want external clients to validate TLS against a stable hostname (SNI) rather than a pod IP.
- As an operator running several clusters on one domain, I want a per-cluster hostname prefix so their shard names don't collide.
Alternatives Considered
Announce an external IP instead of a hostname (cluster-announce-client-ipv4). Valkey supports this, but a LoadBalancer's IP isn't known until the Service is provisioned, which reintroduces a chicken-and-egg problem: you'd have to create the Service, wait for the allocated IP, then CONFIG SET it onto running nodes. A hostname is deterministic at render time and also gives TLS SNI for free, so hostname is the primary path. The API leaves room to add an IP mode later.
One external IP/Service per node instead of per shard. Simpler port math, but it means a lot more LoadBalancers and addresses for a large cluster, and it doesn't match how people think about a cluster ("reach shard N"). Per-shard with one port per node gives the same per-node addressability with far fewer external addresses.
Deterministic, user-chosen NodePort base (for example basePort + index). Predictable for firewall rules, but NodePorts are namespace-global, so two clusters with the same base collide and the second Service is rejected. Letting Kubernetes allocate and reporting the result on status is collision-safe, and the status field covers discoverability.
Operator-managed TLS certificates or DNS records. Out of scope. The operator already uses a bring-your-own-Secret model for TLS and shouldn't own DNS zones. It announces names and applies user-supplied Service annotations (for external-dns and similar); the user owns the certificate SANs and DNS. This could be revisited as a follow-up.
Backward Compatibility
- Breaking change: No.
- Default behavior:
externalAccess is unset by default. With it unset (or enabled: false), the rendered config, the Service set, and the pod template are byte-for-byte identical to today. This is asserted by tests.
- Migration path: None required. Existing clusters are unaffected until they explicitly opt in. Enabling it triggers a single, orchestrated one-at-a-time rolling update (because the announce arguments change the pod template), with the operator's normal proactive failover.
- Disable path: Documented two-step. Set
preferredEndpointType: ip first so clients move off the hostnames, then remove domain or set enabled: false. Doing it in the reverse order would briefly advertise a hostname endpoint that's no longer announced.
- Version floor: The client-facing announce settings require Valkey 9.0+, which is already the operator's documented baseline.
Testing Strategy
- Unit tests: the announce arguments rendered per layer (human-nodename, hostname, client-port including the TLS vs non-TLS branch); the backward-compat gate (nil/disabled renders unchanged); the per-shard Service port layout, NodePort preservation across updates, and NodePort vs LoadBalancer endpoint reporting;
cluster-preferred-endpoint-type rendered only when hostname is selected. A regression test confirms the CLUSTER NODES address parser still extracts the pod IP when an announced hostname is appended, so the reconciler's pod-IP correlation is unaffected.
- envtest: per-shard Service creation with the right selector/type/labels (including
managed-by so the manager cache sees them), a no-op second reconcile with stable ports, the read-back populating status, and scale-in / disable teardown. For Gateway, envtest covers the gated path: with the Gateway API CRDs absent, configuring a Gateway is a no-op that records a warning and still reconciles.
- E2E (kind,
ExternalAccess label): NodePort allocation and status reporting, single-endpoint-per-port resolution, the announced hostname appearing in CLUSTER NODES, and end-to-end external valkey-cli -c access with cross-shard redirects resolving to the announced endpoint.
- E2E (kind,
GatewayAPI label): install the Gateway API CRDs and a Gateway, then assert one TCPRoute per node with the right backend, the ClusterIP default on the backing Service, and route cleanup when the Gateway config is removed.
Open Questions
- Split-horizon DNS.
preferredEndpointType is cluster-wide, so in-cluster clients are also redirected to the hostname. Should the operator help here (for example, an internal-resolution Service), or is documenting split-horizon DNS and accepting the hairpin enough for a first cut?
- Future IP-based client endpoint. Is a
client-ipv4/ipv6 mode alongside hostname worth adding, for environments without convenient DNS? The API is shaped to allow it.
- TLS SAN ergonomics. The first cut documents the SAN contract (the cert must cover the internal Service FQDN plus every shard hostname). Is an optional status warning, for when configured hostnames aren't covered, worth the cert-parsing complexity later?
externalTrafficPolicy: Local preserves the client source IP but requires DNS to target the nodes actually hosting a shard's pods, which the operator can't guarantee with one Service per shard. Document the caveat, or hold off on Local until there's a clean story?- TCPRoute API version. TCPRoute is still a v1alpha2 (experimental-channel) kind in Gateway API v1.5.1 and graduates to v1 in v1.6. The integration pins v1alpha2 for now; should we track v1 as soon as v1.6 ships, or wait for it to settle?
- Operator-managed Gateways and cross-namespace routing. The first cut emits TCPRoutes only, to a same-namespace Gateway the user owns. Is there appetite for the operator to also manage the Gateway/listeners, or to support a Gateway in another namespace (via ReferenceGrant)?
References
Implementation PRs
The feature is delivered as a stack of small, independently reviewable PRs, each building on the previous one. Every PR is backward compatible: a cluster that does not set externalAccess renders exactly as it does today.
Summary
This proposes an
externalAccessconfiguration block on ValkeyCluster that lets clients outside Kubernetes connect to a sharded cluster. The operator exposes each shard through its own Service, optionally announces a stable per-shard hostname, and (when asked) directs clients to those external endpoints using Valkey's client-facing announce settings. Throughout, node-to-node traffic (gossip and replication) stays on internal pod IPs, so the operator's existing reconciliation, which correlatesCLUSTER NODESaddresses with pod IPs, is unaffected. The work is delivered as four small, stacked PRs. Each one is independently useful and fully backward compatible: a cluster that doesn't opt in renders exactly as it does today.Motivation
Today a ValkeyCluster is reachable only from inside the Kubernetes cluster. Every node announces its pod IP, and the only Service is a headless
ClusterIP. Cluster clients follow MOVED/ASK redirects, and in cluster mode those redirects carry the target node's announced address, which is a pod IP that isn't routable from outside. So even if you put a LoadBalancer in front of the pods, the first cross-shard redirect sends the client to an unreachable address and the connection fails.This blocks a real and common deployment: applications running outside the Kubernetes cluster (a different cluster, a VM fleet, on-prem services) that need to use a Valkey cluster hosted in Kubernetes. Users have asked for it, and other Redis/Valkey operators support some form of it.
This need has been raised before: discussion #77 opened the idea (LoadBalancer, Ingress, NodePort, and proxy options, plus TLS termination), and design proposal #259 sketched per-shard Services, reading back allocated ports, publishing endpoints in status, and optional Gateway/TLSRoute SNI routing. This RFC follows that direction and fills in the details: how to keep the cluster bus internal so the reconciler is untouched, how the announce settings are rendered per node, the NodePort allocation and read-back mechanics, and the optional Gateway API path. It starts with TCPRoute rather than TLSRoute/SNI, and leaves TLSRoute as a follow-up.
The core difficulty is that Valkey uses one announced address for both client redirects and the inter-node cluster bus. Naively announcing an external address would push gossip and replication over that external path too, and it would break the operator, which assumes the announced address equals the pod IP. The key enabler is that Valkey 9.0 separates the node-to-node announce settings (
cluster-announce-ip/-port/-bus-port) from the client-facing ones (cluster-announce-hostname,cluster-announce-client-port/-client-tls-port, and the cluster-widecluster-preferred-endpoint-type). That separation is what makes this feasible without disturbing the internal data plane, and 9.0 is already the operator's documented baseline since it's required forshutdown-on-sigterm.Detailed Design
The feature is opt-in through a new
spec.externalAccessblock. When it's absent, nothing changes. When enabled, the operator does four things, which map to the four implementation layers.1. Node identity (human-readable names). With external access on, each node announces
cluster-announce-human-nodenameset to its ValkeyNode name (for examplecluster-sample-1-2), so failure events reference a readable name instead of only the opaque node ID. This rides the same container-argument seam that already sets--cluster-announce-ip. It's a small standalone win and a natural place to introduce the API.2. Per-shard Services. The cluster controller reconciles one Service per shard, selecting that shard's pods. Each Service exposes one port per node, and each port is pinned to a single pod by giving the server container a node-unique port name (
vk-n<idx>) and targeting it by name. Kubernetes only adds a pod to a Service port's endpoints if the pod declares a container port with that name, so the port lands on exactly one pod. This is what lets a client reach a specific primary or replica, which cluster redirects need.For
NodePort(the default), the operator does not assign port numbers. It lets Kubernetes allocate them, which is the only way to avoid collisions across clusters in a namespace, then reads the allocated ports back and records them on the cluster status. ForLoadBalancer, each shard has its own external address, so the frontend ports are simply6379 + nodeIndex.3. Per-shard hostnames. With a
domainset, each shard announcescluster-announce-hostname = <hostnamePrefix>-<shardIndex>.<domain>(prefix defaults toshard). This is metadata only. It appears inCLUSTER SLOTSand is useful for TLS SNI, but it doesn't change where clients are sent yet. The cluster bus keeps using pod IPs.4. Client endpoint selection. Each node announces its external client port (read back from the shard Service in step 2) via
cluster-announce-client-port, orcluster-announce-client-tls-portwhen TLS is enabled. SettingpreferredEndpointType: hostnamethen flipscluster-preferred-endpoint-typecluster-wide, so MOVED/ASK andCLUSTER SLOTShand clients the shard hostname and external port. A client connecting to any shard now gets correct cross-shard redirects to a routablehostname:port.5. Gateway API exposure (optional). As an alternative to NodePort/LoadBalancer Services, the operator can expose the cluster through the Gateway API. With
externalAccess.gatewayset, it creates one TCPRoute per node, attaching to a Gateway the user already runs and forwarding to the node's port on the per-shard Service. The Gateway is then the only external surface, so the backing Service defaults to ClusterIP. The announced client port is the Gateway listener port,basePort + shardIndex*(replicas+1) + nodeIndex, which is deterministic and needs no read-back. Gateway API is an optional dependency: the operator checks at startup whether the TCPRoute kind is served, and only then registers the watch and reconcile path. On a cluster without the Gateway API CRDs, it behaves exactly as it would without Gateway support, and never references a type the API server can't serve.Components:
ExternalAccessSpeconValkeyClusterSpec, mirrored ontoValkeyNodeSpec; aShardEndpointstatus type onValkeyClusterStatus.reconcileShardServicesstep (Services first, before nodes, so the allocated ports are available), endpoint read-back into status, per-node port resolution, and scale-in Service cleanup.cluster-preferred-endpoint-type) added to the shared base config so it rolls through the existing config-hash path.Reconcile ordering and rollout. Services are reconciled before nodes, so a NodePort is normally allocated and read back on the same pass that builds the node. The announced values live in the pod template, so a change flows through the operator's existing one-at-a-time, proactive-failover roll. No new rollout machinery is needed. The cluster-wide endpoint-type setting rolls through the existing config hash.
User workflow. Set
externalAccess.enabled: truewith aserviceType, and the operator creates the Services and reports the external ports understatus.externalEndpoints. Add adomainto announce hostnames, create DNS records pointing those hostnames at the Services, then setpreferredEndpointType: hostnameto direct clients to them.API Changes
CRD validation is expressed as CEL, since the repo has no validating webhook:
hostnamePrefixmust be a DNS label, anddomainis required whenpreferredEndpointTypeishostname. NodePort numbers are not user-settable; Kubernetes allocates them.User Stories
Alternatives Considered
Announce an external IP instead of a hostname (
cluster-announce-client-ipv4). Valkey supports this, but a LoadBalancer's IP isn't known until the Service is provisioned, which reintroduces a chicken-and-egg problem: you'd have to create the Service, wait for the allocated IP, thenCONFIG SETit onto running nodes. A hostname is deterministic at render time and also gives TLS SNI for free, so hostname is the primary path. The API leaves room to add an IP mode later.One external IP/Service per node instead of per shard. Simpler port math, but it means a lot more LoadBalancers and addresses for a large cluster, and it doesn't match how people think about a cluster ("reach shard N"). Per-shard with one port per node gives the same per-node addressability with far fewer external addresses.
Deterministic, user-chosen NodePort base (for example
basePort + index). Predictable for firewall rules, but NodePorts are namespace-global, so two clusters with the same base collide and the second Service is rejected. Letting Kubernetes allocate and reporting the result on status is collision-safe, and the status field covers discoverability.Operator-managed TLS certificates or DNS records. Out of scope. The operator already uses a bring-your-own-Secret model for TLS and shouldn't own DNS zones. It announces names and applies user-supplied Service annotations (for external-dns and similar); the user owns the certificate SANs and DNS. This could be revisited as a follow-up.
Backward Compatibility
externalAccessis unset by default. With it unset (orenabled: false), the rendered config, the Service set, and the pod template are byte-for-byte identical to today. This is asserted by tests.preferredEndpointType: ipfirst so clients move off the hostnames, then removedomainor setenabled: false. Doing it in the reverse order would briefly advertise a hostname endpoint that's no longer announced.Testing Strategy
cluster-preferred-endpoint-typerendered only when hostname is selected. A regression test confirms theCLUSTER NODESaddress parser still extracts the pod IP when an announced hostname is appended, so the reconciler's pod-IP correlation is unaffected.managed-byso the manager cache sees them), a no-op second reconcile with stable ports, the read-back populating status, and scale-in / disable teardown. For Gateway, envtest covers the gated path: with the Gateway API CRDs absent, configuring a Gateway is a no-op that records a warning and still reconciles.ExternalAccesslabel): NodePort allocation and status reporting, single-endpoint-per-port resolution, the announced hostname appearing inCLUSTER NODES, and end-to-end externalvalkey-cli -caccess with cross-shard redirects resolving to the announced endpoint.GatewayAPIlabel): install the Gateway API CRDs and a Gateway, then assert one TCPRoute per node with the right backend, the ClusterIP default on the backing Service, and route cleanup when the Gateway config is removed.Open Questions
preferredEndpointTypeis cluster-wide, so in-cluster clients are also redirected to the hostname. Should the operator help here (for example, an internal-resolution Service), or is documenting split-horizon DNS and accepting the hairpin enough for a first cut?client-ipv4/ipv6mode alongside hostname worth adding, for environments without convenient DNS? The API is shaped to allow it.externalTrafficPolicy: Localpreserves the client source IP but requires DNS to target the nodes actually hosting a shard's pods, which the operator can't guarantee with one Service per shard. Document the caveat, or hold off onLocaluntil there's a clean story?References
valkey.conf, "CLUSTER DOCKER/NAT support"): https://github.com/valkey-io/valkey/blob/9.0.0/valkey.confCLUSTER SLOTS/ preferred endpoint type: https://valkey.io/commands/cluster-slots/Implementation PRs
The feature is delivered as a stack of small, independently reviewable PRs, each building on the previous one. Every PR is backward compatible: a cluster that does not set
externalAccessrenders exactly as it does today.externalAccessAPI block (initially justenabled) and, as the first behaviour, announcescluster-announce-human-nodenameso cluster events reference the ValkeyNode name instead of only the opaque node ID. No networking yet.status.externalEndpoints. AddsserviceType,externalTrafficPolicy, andserviceAnnotations.hostnamePrefixanddomain, announcing<hostnamePrefix>-<shardIndex>.<domain>per shard. This is metadata only (useful forCLUSTER SLOTSand TLS SNI) and does not yet change where clients are sent.preferredEndpointTypeand the per-node client-port announce (cluster-announce-client-port/-client-tls-port). Settinghostnameflipscluster-preferred-endpoint-typecluster-wide so clients are redirected to the shard hostname and external port. This completes the Service-based client-routing path.