g3proxy feature proposal

[zh-jq]

Enhanments

1. Support Post-Quantum Cryptography algorithms and enable by default

2. Support RFC 9209 Proxy-Status HTTP Response Header Field

Refactor

1. Support type specific metrics for servers/escapers

   Add a new EmitStats trait, with a emit method, and impl it for all server & escaper stats.

   Then remove ServerStats and EscaperStats traits, and just store ArcEmitStats in server & escaper stats registry.

2. Add a new rust-variant-ssl rustls backend and replace ring

Breaking Changes

A major version bump will be needed for the following changes:

1. Chain proxy-* escapers on top of other escapers

   Remove the tcp-connect and udp-connect code in proxy-* escapers, and let them use the corresponding tcp/udp methods of the next escaper.

   This will remove a lot of code, and will allow the chaining of multiple proxies at one place.

   The disadvantage is that the returned tcp/udp socket in wrapped in Box, which may decrease the performance a bit.

Big new ones

A minor version bump will be needed for the following changes:

1. Support remote user authentication

   • Password based such as LDAP / Radius

   • Kerberos

2. Add reverse proxy connection support

   Support ngrok/frp like features

   • Allow to set direct-float escaper to harbor mode, it will collect client connections and use them when needed.

   • Allow port servers to set harbor config field, they will send the client connections directly to that harbor mode escaper.

   • Add an anchor-tcp server, it will act as a client and pool connect via it's escaper to a remote g3proxy harbor. When new connection request from the remote harbor, it will open new connections via it's next server / escaper.

   • Add an anchor-quic server, which pool connect to the remote harbor in QUIC.

3. Support cluster level rate limitation and speed limitation

4. Support H2 http proxy

5. Support RFC 9298 Proxying UDP in HTTP

6. Use EBPF to hot-upgrade REUSEPORT sockets gracefully on Linux

   This is required to hot upgrade UDP sockets.

7. Support H3 http proxy

[DanielHaimanot]

Re: Stats Refactor

I started some initial work looking into refactoring the proxy stats code. In addition to simplifying type specific stats implementations; I'm thinking about introducing a metrics facade, and recorder impl for different collectors (statd, prom,etc) to help make it easier to support different user export preferences. We can discuss this more.

Re: Add reverse proxy connection support

I think this may also be useful to improve tcp connection pool reuse between chained proxies in simple forwarding scenarios. So that we can re-use intermediate proxy connections between downstream clients.

[zh-jq]

Re: Stats Refactor

I started some initial work looking into refactoring the proxy stats code. In addition to simplifying type specific stats implementations; I'm thinking about introducing a metrics facade, and recorder impl for different collectors (statd, prom,etc) to help make it easier to support different user export preferences. We can discuss this more.

The original design is that we use statsd (maybe g3statsd or any other statsd implementations) to support different metrics collectors (influxdb/opentsdb/graphite/prometheus). Though it's possible to support prometheus directly in g3proxy, but I don't see enough benefits.

Re: Add reverse proxy connection support

I think this may also be useful to improve tcp connection pool reuse between chained proxies in simple forwarding scenarios. So that we can re-use intermediate proxy connections between downstream clients.

Yes custom protocols on top of TCP/QUIC can be designed for the reverse proxy connection.

[DanielHaimanot]

Re: Stats Refactor
I started some initial work looking into refactoring the proxy stats code. In addition to simplifying type specific stats implementations; I'm thinking about introducing a metrics facade, and recorder impl for different collectors (statd, prom,etc) to help make it easier to support different user export preferences. We can discuss this more.

The original design is that we use statsd (maybe g3statsd or any other statsd implementations) to support different metrics collectors (influxdb/opentsdb/graphite/prometheus). Though it's possible to support prometheus directly in g3proxy, but I don't see enough benefits.

Right, it's a relatively minor improvement for certain scenarios. For cloud-native deployments it would help simplify getting started as most users already familiar with proxies such as Envoy, Traefik etc would not expect an additional binary for observability. Although prometheus has real downsides over statsd, it is popular with K8s.

[DanielHaimanot]

Re: Support H2 http proxy

I know we already have ALPN H2 upgrades implemented; is the intention for H2 http proxy feature to implement a new type for h2c?

Following up on this, there might be some benefit for us to improve integration test conformance for H2/H3 implementations against the standard model (eg nghttp2)

[zh-jq]

For cloud-native deployments it would help simplify getting started as most users already familiar with proxies such as Envoy, Traefik etc would not expect an additional binary for observability. Although prometheus has real downsides over statsd, it is popular with K8s.

Yes prometheus make sense for k8s. We also need to design a way to expose HTTP APIs, including but not limited to prometheus, more APIs can be added.

[zh-jq]

Re: Support H2 http proxy

I know we already have ALPN H2 upgrades implemented; is the intention for H2 http proxy feature to implement a new type for h2c?

No. Only the interception code have H2 support for now.

Chromium have support for https proxy, both h1 and h2. The http_proxy server may be extended to support H2 if ALPN matched, but I'm not sure whether we need to support requests other than extended CONNECT (used in Websocket / MASQUE).

[DanielHaimanot]

Re: Support H2 http proxy
I know we already have ALPN H2 upgrades implemented; is the intention for H2 http proxy feature to implement a new type for h2c?

No. Only the interception code have H2 support for now.

Chromium have support for https proxy, both h1 and h2. The http_proxy server may be extended to support H2 if ALPN matched, but I'm not sure whether we need to support requests other than extended CONNECT (used in Websocket / MASQUE).

Gotcha, that makes sense. It will be nice to have H3/MASQUE support.

[DanielHaimanot]

For transparency should we organize proposed features into separate issues? Also for some of the larger feature does having an optional designs folder --link make sense? I feel like it might make it easier for potential contributors to get on the same page about a proposed design change. wdyt?

[zh-jq]

For transparency should we organize proposed features into separate issues?

Yes I think so. It will be good for detailed discussion for the big ones.

Also for some of the larger feature does having an optional designs folder --link make sense? I feel like it might make it easier for potential contributors to get on the same page about a proposed design change. wdyt?

I like it but it will be a bit hard. I definitely need a lot of help on this. AI seems to be helpful here to write designs for the existed code.