Skip to content

reverseproxy: re-apply WebSocket header normalization after header ops#7786

Open
sapirbaruch wants to merge 7 commits into
caddyserver:masterfrom
sapirbaruch:fix-websocket-header-normalization
Open

reverseproxy: re-apply WebSocket header normalization after header ops#7786
sapirbaruch wants to merge 7 commits into
caddyserver:masterfrom
sapirbaruch:fix-websocket-header-normalization

Conversation

@sapirbaruch

@sapirbaruch sapirbaruch commented Jun 1, 2026

Copy link
Copy Markdown

Fixes #7784.

When transport or headers ops are configured on a reverse proxy handler, proxyLoopIteration rebuilds r.Header from scratch using copyHeader. Because copyHeader uses http.Header.Add internally, Go's CanonicalHeaderKey canonicalizes the header names — this lowercases the second word in Sec-WebSocket-*, so Sec-WebSocket-Key becomes Sec-Websocket-Key.

normalizeWebsocketHeaders is already called in prepareRequest to fix this exact problem, but the subsequent header rebuild in the retry loop undoes it. Adding the same call after the rebuild keeps the headers in RFC 6455-compliant form (Sec-WebSocket-* with uppercase S) regardless of whether header ops are configured.

The call is cheap and safe for non-WebSocket requests: normalizeWebsocketHeaders iterates a five-element map and only mutates headers that actually exist, so it's effectively a no-op when there are no Sec-WebSocket-* headers present.


AI assistance disclosure: This PR was developed with the assistance of an AI tool (Claude), per the project's contribution guidelines.

When transport or user header ops are configured, proxyLoopIteration
rebuilds r.Header from scratch using copyHeader. copyHeader calls
http.Header.Add internally which canonicalizes header names via
CanonicalHeaderKey — this lowercases the 'S' in WebSocket, turning
"Sec-WebSocket-Key" into "Sec-Websocket-Key".

normalizeWebsocketHeaders was already called in prepareRequest to fix
this, but the subsequent header rebuild in proxyLoopIteration undoes
it. Calling normalizeWebsocketHeaders again after the rebuild restores
the RFC 6455-compliant casing for all Sec-WebSocket-* headers.

Fixes caddyserver#7784
@CLAassistant

CLAassistant commented Jun 1, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@mholt

mholt commented Jun 2, 2026

Copy link
Copy Markdown
Member

Can you the human please sign the CLA please @sapirbaruch ?

@mholt

mholt commented Jun 2, 2026

Copy link
Copy Markdown
Member

Also would your LLM be able to add a test?

@francislavoie francislavoie added the bug 🐞 Something isn't working label Jun 2, 2026
@francislavoie francislavoie added this to the v2.11.4 milestone Jun 2, 2026
Add TestNormalizeWebsocketHeaders covering the four cases:
- canonical (lowercased) header names are renamed to RFC 6455 form
- headers already in the correct form are left unchanged
- non-WebSocket headers are untouched
- empty header map is a no-op

Add TestNormalizeWebsocketHeadersSurvivesCopyHeader as a targeted
regression test for caddyserver#7784: simulates the header-rebuild that
proxyLoopIteration performs when transport or header ops are
configured, verifies that calling normalizeWebsocketHeaders afterwards
restores Sec-WebSocket-* to the RFC 6455 casing.
@mholt

mholt commented Jun 2, 2026

Copy link
Copy Markdown
Member

My last question before we merge it, that I just considered, is should we have Go fix CanonicalHeaderKey instead of us hacking around it? If the canonical form is WebSocket maybe it shouldn't be changing it to Websocket.

@sapirbaruch

Copy link
Copy Markdown
Author

The Go team explicitly closed this as won't-fix back in 2016 (golang/go#18495). Brad Fitzpatrick's response:

RFC 6455 can't mandate the case of headers, since HTTP/1 says that they're case insensitive. And in HTTP/2 there is no case on the wire. I don't want to complicate Go and encourage buggy libraries from assuming case.

So CanonicalHeaderKey will never special-case WebSocket headers — the normalization has to live in Caddy. The normalizeWebsocketHeaders helper (introduced in #6621) is the right layer for this, and the fix just makes sure it's called after the header rebuild in the retry loop, not only before it.

http.Header.Get re-canonicalizes its argument via CanonicalHeaderKey,
so rebuilt.Get("Sec-WebSocket-Key") looks up the Go-canonical form
"Sec-Websocket-Key" — the key normalizeWebsocketHeaders just deleted.
Use a direct map lookup instead to assert the RFC 6455 key is present.
Comment thread modules/caddyhttp/reverseproxy/reverseproxy.go Outdated
@WeidiDeng

Copy link
Copy Markdown
Member

AI disclosure is required.

@mholt mholt modified the milestones: v2.11.4, v2.11.5 Jun 3, 2026
@mholt

mholt commented Jun 3, 2026

Copy link
Copy Markdown
Member

Thanks for the reminder @WeidiDeng . Yes, the PR is missing our standard assistance disclosure. It's required even if not using AI, but I can tell this one is, and I think in my head since I could tell I forgot to check for the disclosure.

@tejgokani

Copy link
Copy Markdown

@sapirbaruch — apologies for the duplicate (#7803). I missed your PR when scanning. Closing mine now.

If useful, two things from my work that might be worth folding in:

1. A regression test that exercises the rebuild path itself. Your unit tests on normalizeWebsocketHeaders directly are clean, but they don't fail if the new call at the rebuild site is later removed. A test that goes through the rebuild block (via a small helper extraction or via proxyLoopIteration setup) catches that. Happy to share the diff; the gist is a table-driven test with three cases — user header_up only, the HTTPS transport-injected Host op only, and both together — that constructs a Handler{}, calls into the rebuild path, and asserts the RFC 6455 casing reaches r.Header. I verified it fails without the fix and passes with it.

2. A raw-net.Listen upstream for wire-level evidence. Go's net/http server canonicalizes incoming header keys on receive, so an application-layer upstream hides the bug even when it's present on the wire. A ~30-line raw TCP upstream that prints the literal request bytes shows the difference clearly:

PRE-FIX:   Sec-Websocket-Key: dGhlIHNhbXBsZSBub25jZQ==
POST-FIX:  Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==

Useful for the PR body / future regression debugging.

Totally fine to skip either of these — just offering in case they save you time. Thanks for the citation on golang/go#18495, that was the right context I was missing.


Assistance disclosure: minimal AI assistance — used Claude (Opus 4.7) to compare implementation shapes and draft this comment. Authored and verified by me.

@steadytao

Copy link
Copy Markdown
Member

I added the rebuild-path regression test locally since this PR has gone stale. The updated test exercises the actual header rebuild path for user header ops, the transport-injected Host op case and both together. It fails if the post-rebuild normalizeWebsocketHeaders call is removed so it should cover the regression Tej pointed out.

Comment thread modules/caddyhttp/reverseproxy/reverseproxy.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug 🐞 Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WebSocket header casing normalization reverted when header ops are configured

7 participants