KEP-5695: kubectl reverse port-forward #5697

willianpaixao · 2025-11-18T19:14:22Z

What type of PR is this?
/kind kep

What this PR does / why we need it:
Adds KEP for reverse port-forwarding in kubectl, addressing long-standing issue kubernetes/kubernetes#20227.

Which issue(s) this PR fixes:
Fixes #5695
Addresses kubernetes/kubernetes#20227

Special notes for your reviewer:
This KEP proposes adding `--reverse` flag to `kubectl port-forward` to enable
reverse tunneling (pod → local machine), similar to SSH's `-R` flag.

This has been a requested feature since 2016 and received a PoC implementation
in 2017 (PR kubernetes/kubernetes#57320) that was deferred pending SPDY deprecation.

/sig cli
/sig node

Add reverse port-forwarding capability to kubectl, enabling pods to connect to services running on the developer's local machine. This addresses kubernetes/kubernetes#20227 which has been open since 2016 with significant community interest. Signed-off-by: Willian Paixao <[email protected]>

linux-foundation-easycla · 2025-11-18T19:14:30Z

The committers listed above are authorized under a signed CLA.

✅ login: willianpaixao / name: willian.eth (1b0dd3c, 3799c78, a34e69d, ece5dd7)

k8s-ci-robot · 2025-11-18T19:14:31Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: willianpaixao
Once this PR has been reviewed and has the lgtm label, please assign soltysh for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/sig-cli/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-11-18T19:14:32Z

Welcome @willianpaixao!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-11-18T19:14:32Z

Hi @willianpaixao. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copilot

Pull Request Overview

This PR introduces KEP-5695, which proposes adding reverse port-forwarding capability to kubectl port-forward. This feature would enable containers in Kubernetes pods to connect back to services running on a developer's local machine, similar to SSH's -R flag. This addresses a long-standing feature request from 2016 (issue #20227).

Adds a new --reverse flag to kubectl port-forward for reverse tunneling
Introduces protocol extensions to support bidirectional streaming between pods and local machines
Includes comprehensive graduation criteria from alpha (v1.34) through GA (v1.36)

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File	Description
keps/sig-cli/5695-kubectl-reverse-port-forward/kep.yaml	Defines KEP metadata including feature gates (KubectlReversePortForward), milestones, and basic metrics configuration
keps/sig-cli/5695-kubectl-reverse-port-forward/README.md	Comprehensive KEP documentation covering motivation, design details, test plans, graduation criteria, and production readiness considerations

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

keps/sig-cli/5695-kubectl-reverse-port-forward/kep.yaml

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

- Detail required unit tests and existing test infrastructure. - Update E2E test requirements. - Update Graduation Criteria to include enabling existing test stubs. - Update Implementation History with test infrastructure preparation. Signed-off-by: Willian Paixao <[email protected]>

Signed-off-by: Willian Paixao <[email protected]>

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

willianpaixao · 2025-11-20T13:41:13Z

@aojea all your comments, as well as Copilot's, were addressed. Thank you for the review, happy to keep the discussion open.

Copilot

Pull Request Overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-20T13:45:48Z

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

+    - Detection: New connection attempts fail with error: `reverse port-forward connection limit reached`
+    - Mitigations: This is intentional rate limiting to prevent resource exhaustion. User can
+      wait for existing connections to close, or adjust the limit if appropriate for their use case.
+    - Diagnostics: Metric `kubectl_portforward_reverse_connections_rejected_total{reason="limit"}` will increment.


The metric name referenced here (kubectl_portforward_reverse_connections_rejected_total) is not defined in the metrics list in kep.yaml (lines 44-49) or in the SLI section (lines 526-530). If this metric will be collected, it should be added to the metrics list in kep.yaml.

Copilot · 2025-11-20T13:45:48Z

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

+kubectl port-forward pod-name 8080:80
+
+# New reverse port-forward syntax
+kubectl port-forward --reverse pod-name 8080:80
+# or
+kubectl port-forward -R pod-name 8080:80
+
+# Multiple ports (both forward and reverse)
+kubectl port-forward --reverse pod-name 8080:80 9090:9000
+```
+
+**Port Specification Semantics:**
+- Forward mode (existing): `LOCAL_PORT:REMOTE_PORT` - local port forwards to remote port
+- Reverse mode (new): `REMOTE_PORT:LOCAL_PORT` - remote port forwards to local port


Potential confusion in port specification semantics: The documentation states that in reverse mode REMOTE_PORT:LOCAL_PORT means "remote port forwards to local port", but this contradicts the user story examples where 8080:8080 is used symmetrically.

For example, line 132 shows kubectl port-forward --reverse mypod 8080:8080 with the comment "Those connections get tunneled back to your local machine's port 8080", which suggests the format is still LOCAL:REMOTE or at least ambiguous.

Please clarify the port specification format consistently throughout the document. If it truly reverses to REMOTE:LOCAL (like SSH -R), make this very clear in all examples and add a warning about the semantic change.

Suggested change

kubectl port-forward pod-name 8080:80

# New reverse port-forward syntax

kubectl port-forward --reverse pod-name 8080:80

# or

kubectl port-forward -R pod-name 8080:80

# Multiple ports (both forward and reverse)

kubectl port-forward --reverse pod-name 8080:80 9090:9000

```

**Port Specification Semantics:**

- Forward mode (existing): `LOCAL_PORT:REMOTE_PORT` - local port forwards to remote port

- Reverse mode (new): `REMOTE_PORT:LOCAL_PORT` - remote port forwards to local port

# Format: LOCAL_PORT:REMOTE_PORT (local port forwards to remote port in the pod)

kubectl port-forward pod-name 8080:80

# New reverse port-forward syntax

# Format: REMOTE_PORT:LOCAL_PORT (remote port in the pod forwards to local port on your machine)

kubectl port-forward --reverse pod-name 8080:80

# or

kubectl port-forward -R pod-name 8080:80

# Multiple ports (reverse mode)

# Format: REMOTE_PORT1:LOCAL_PORT1 REMOTE_PORT2:LOCAL_PORT2 ...

kubectl port-forward --reverse pod-name 8080:80 9090:9000

⚠️ WARNING:
In reverse mode (--reverse or -R), the port specification format is reversed compared to forward mode.

Forward mode: LOCAL_PORT:REMOTE_PORT (local → pod)

Reverse mode: REMOTE_PORT:LOCAL_PORT (pod → local)
This is similar to SSH's -R flag, but can be confusing if you are used to the default kubectl port-forward syntax.
Double-check your port order when using reverse mode to avoid unexpected behavior!

Port Specification Semantics:

Forward mode (existing): LOCAL_PORT:REMOTE_PORT — local port forwards to remote port in the pod

Reverse mode (new): REMOTE_PORT:LOCAL_PORT — remote port in the pod forwards to local port on your machine

To be very honest I went back-and-forth with that in my mind while writing that, so it might be inconsistent.
What would be a good pattern?

Copilot · 2025-11-20T13:45:48Z

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

+**4. Container Runtime Changes** (External repositories):
+
+Changes required in container runtimes to support reverse port-forwarding:
+
+**containerd** (`containerd/containerd`):
+- Modify streaming server ([`internal/cri/streamingserver/server.go`](https://github.com/containerd/containerd/blob/main/internal/cri/streamingserver/server.go))
+- Add reverse port-forward handler endpoint
+- Implement network namespace listener creation
+- Handle bidirectional streaming for reverse connections
+
+**cri-o** (`cri-o/cri-o`):
+- Modify port forward implementation ([`server/container_portforward.go`](https://github.com/cri-o/cri-o/blob/main/server/container_portforward.go))
+- Add reverse mode support to port forward handler
+- Implement listener in container network namespace
+- Stream reverse connections back to kubelet
+
+**Note**: This KEP requires coordinated changes across multiple projects (kubernetes, containerd, cri-o). Implementation will need agreement and collaboration from runtime maintainers.


Inconsistency: The flow diagram shows that kubelet creates the listener in the pod's network namespace (line 272), but the Implementation Overview section states this will be done using "native Go listeners" in the kubelet (line 230). However, the container runtime changes section (lines 240-256) indicates that containerd and cri-o need to implement this functionality. Please clarify which component is actually responsible for creating the listener in the pod's network namespace - is it kubelet or the container runtime?

Opsie, yeah, it makes sense, I'll take a closer look at it later.

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

keps/sig-cli/5695-kubectl-reverse-port-forward/kep.yaml

lmktfy · 2025-11-20T22:55:52Z

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

+
+#### Story 3: Database Migration from Local Tools
+
+As a database administrator, I want to run database migration tools from my local machine against a database pod without exposing the database outside the cluster, using a secure reverse tunnel.


nit: clarify that this is a schema update, not a data move

lmktfy · 2025-11-20T22:56:36Z

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

+
+1. **Security Consideration**: Reverse port-forwarding exposes local services to remote pods. Users must understand the security implications and only use this feature in development/debugging scenarios.
+
+2. **Network Namespace**: The listening port is created in the pod's network namespace, accessible only to processes within that pod (not to other pods in the cluster).


Are we assuming that the Pod is running on a Linux node? Windows doesn't have network namespaces.

I personally have zero knowledge on Windows implementation, would need help on that to expand the KEP.

lmktfy · 2025-11-20T22:58:05Z

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

+- Handle incoming pod connections and stream them to kubectl
+- Clean up listeners when kubectl disconnects
+
+**3. Protocol Changes** (`pkg/kubelet/cri/streaming/portforward/`):


If I want to deny reverse proxy and allow forward proxy, will I be able to write a policy (eg Rego) to enforce this?

The answer should be Yes.

Also good point, I should clarify that. Thanks.

lmktfy · 2025-11-20T22:59:00Z

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

+  - Connection refused on local port
+  - Network namespace access failures
+  - Stream protocol errors
+


What if the container runtime doesn't support this kind of proxying?

@aojea raised this point so I've started to dig and create issues in the respective projects.

lmktfy · 2025-11-20T23:00:09Z

keps/sig-cli/5695-kubectl-reverse-port-forward/kep.yaml

+# The following PRR answers are required at alpha release
+# List the feature gate name and the components for which it must be enabled
+feature-gates:
+  - name: KubectlReversePortForward


This name implies that the client is kubectl. It can be but doesn't have to be. I would rename the feature gate.

Good point, I didn't consider other clients. Before renaming, do you think this KEP belongs to SIG-CLI or SIG-NODE?

lmktfy · 2025-11-20T23:00:46Z

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

+
+- **Step 8**: Verify pod and network connectivity:
+
+```bash


nit: the script is not valid bash

lmktfy · 2025-11-20T23:00:57Z

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

+- **Step 9**: If all else fails, disable the feature and use forward port-forward
+  to verify base functionality:
+
+```bash


nit: the script is not valid bash

I'm sorry, can you be more specific? The syntax is valid.
Is it because I've omitted the launch of the nginx container?

lmktfy · 2025-11-20T23:01:31Z

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md

+Extend API server to proxy connections between pods and external endpoints.
+
+**Rejected because**: API server should not be in the data path for debugging tools. This is a kubectl/kubelet concern.
+


You could also launch an ephemeral container to achieve this aim.

Copilot AI review requested due to automatic review settings November 18, 2025 19:14

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/cli Categorizes an issue or PR as relevant to SIG CLI. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Nov 18, 2025

github-project-automation bot added this to SIG CLI Nov 18, 2025

github-project-automation bot moved this to Needs Triage in SIG CLI Nov 18, 2025

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 18, 2025

k8s-ci-robot requested a review from ardaguclu November 18, 2025 19:14

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Nov 18, 2025

k8s-ci-robot requested a review from mpuckett159 November 18, 2025 19:14

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 18, 2025

Copilot started reviewing on behalf of willianpaixao November 18, 2025 19:14 View session

Copilot finished reviewing on behalf of willianpaixao November 18, 2025 19:16

Copilot AI reviewed Nov 18, 2025

View reviewed changes

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Nov 18, 2025

willianpaixao added 2 commits November 19, 2025 11:58

KEP-5695: Add troubleshooting section inspired from KEP-4006

3799c78

Signed-off-by: Willian Paixao <[email protected]>

aojea reviewed Nov 19, 2025

View reviewed changes

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md Outdated Show resolved Hide resolved

aojea reviewed Nov 19, 2025

View reviewed changes

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md Outdated Show resolved Hide resolved

aojea reviewed Nov 19, 2025

View reviewed changes

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md Outdated Show resolved Hide resolved

aojea reviewed Nov 19, 2025

View reviewed changes

keps/sig-cli/5695-kubectl-reverse-port-forward/README.md Show resolved Hide resolved

KEP-5695: addressing comments by reviewers

a34e69d

willianpaixao requested a review from Copilot November 20, 2025 13:42

Copilot started reviewing on behalf of willianpaixao November 20, 2025 13:43 View session

Copilot finished reviewing on behalf of willianpaixao November 20, 2025 13:44

Copilot AI reviewed Nov 20, 2025

View reviewed changes

willianpaixao mentioned this pull request Nov 20, 2025

Add reverse port forwarding support cri-o/cri-o#9611

Open

lmktfy reviewed Nov 20, 2025

View reviewed changes


		#### Story 3: Database Migration from Local Tools

		As a database administrator, I want to run database migration tools from my local machine against a database pod without exposing the database outside the cluster, using a secure reverse tunnel.


		1. Security Consideration: Reverse port-forwarding exposes local services to remote pods. Users must understand the security implications and only use this feature in development/debugging scenarios.

		2. Network Namespace: The listening port is created in the pod's network namespace, accessible only to processes within that pod (not to other pods in the cluster).

		Extend API server to proxy connections between pods and external endpoints.

		Rejected because: API server should not be in the data path for debugging tools. This is a kubectl/kubelet concern.

KEP-5695: kubectl reverse port-forward #5697

Are you sure you want to change the base?

KEP-5695: kubectl reverse port-forward #5697

Conversation

willianpaixao commented Nov 18, 2025

Uh oh!

linux-foundation-easycla bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Nov 18, 2025

Uh oh!

k8s-ci-robot commented Nov 18, 2025

Uh oh!

k8s-ci-robot commented Nov 18, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

willianpaixao commented Nov 20, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

linux-foundation-easycla bot commented Nov 18, 2025 •

edited

Loading