Support async/fire-and-forget enforcement off the critical path

[davidxia]

kgateway version

2.3.2

Is your feature request related to a problem? Please describe.

I want global rate limiting but don't want every incoming request to result in a call to the rate limiting service which will add latency to each request. I want asynchronous rate limiting (sometimes called async-evaluate or post-evaluate) where the proxy lets the request through while simultaneously asking the rate limit service for a decision. If the limit is hit, only the next request is blocked.

I read this is how companies like Cloudflare implement their rate limiting system.

This is why the increment jobs are run asynchronously without slowing down the requests. If the request rate is above the threshold, another piece of data is stored asking all servers in the PoP to start applying the mitigation for that client. Only this bit of information is checked during request processing.

Kgateway's global rate limiting doc suggests the only mode is synchronous.

3. The Gateway extracts descriptors and sends them to the Rate Limit Service.
4. The Rate Limit Service applies configured limits for those descriptors and returns a decision to the Gateway.
5. If allowed, the Gateway forwards the request to the App.
6. If the rate limit is reached, the Gateway denies the request and returns a message to the Client.

Describe the solution you'd like

Envoy's global HTTP rate-limit filter supports apply_on_stream_done on route/vhost-level
RateLimit actions (envoy.config.route.v3.RateLimit.apply_on_stream_done), which sends the
rate-limit-service call after the stream completes instead of gating the request:

A rate limit applied on the stream completion is "fire-and-forget" by nature, and rate limit
is not enforced by this config. In other words, the current request won't be blocked when this
is true, but the budget will be updated for the subsequent requests based on the action with
this field set to true.

This removes the RLS round trip from the request's critical path — the current request proceeds
immediately, and only a later request can be denied once the budget is exceeded.

kgateway's TrafficPolicy.spec.rateLimit.global (RateLimitPolicy in
api/v1alpha1/kgateway/traffic_policy_types.go) currently only exposes descriptors and
extensionRef, with no way to set this field. Requesting it be surfaced on
RateLimitDescriptor/RateLimitPolicy so users can trade strict enforcement for latency
(e.g. combining a synchronous descriptor for a hard ceiling with an apply_on_stream_done
descriptor for cost/usage-based accounting).

Describe alternatives you've considered

No response

Additional Context

Related: this is distinct from #6540 (shadow mode), which never enforces at all rather than
deferring enforcement to a subsequent request.

[jbohanon]

I want to clarify the intent here... Rate limits configured with this flag never block requests by design. AFAICT they're mostly for external tracking. The documentation explicitly states this. So you would need a separate RateLimitPolicy that would still require a round trip to apply the block. Please correct me if I am misunderstanding what you are trying to achieve, but there is no way to use this flag to remove round-tripping the RLS for enforcement.

Ref the field documentation: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto.html#envoy-v3-api-field-config-route-v3-ratelimit-apply-on-stream-done

The crux of where I feel I may be misunderstanding your proposed use case is here:

deferring enforcement to a subsequent request

If every request is deferring to a subsequent request, then no request is ever enforced? Enforcement eventually requires a RT to RLS

[davidxia]

Hi, @jbohanon thanks for asking. I'm trying to achieve the low-latency, async architecture that Cloudflare and Uber have described.

This is why the increment jobs are run asynchronously without slowing down the requests. If the request rate is above the threshold, another piece of data is stored asking all servers in the PoP to start applying the mitigation for that client. Only this bit of information is checked during request processing. link

Enforce per-request decisions locally based on directives received from aggregators... Aggregate zone data to determine global utilization and push updated drop ratio directives back down to aggregators and clients. link

deferring enforcement to a subsequent request

By this I mean that the previous request has triggered the rate limiter to push a directive to the local edge proxy to limit subsequent matching requests. So the next request is rejected without a round trip to the rate limiter.

I'm not super familiar with the Envoy and kgateway functionality here so I might be misunderstanding some stuff or the feasibility. Happy to provide more details!

[jbohanon]

Cool! Since you're talking about decoupling processing from enforcement anyway, have you considered using external processing for receiving the request data you care about (headers, body, trailers, all, some, etc)?

By this I mean that the previous request has triggered the rate limiter to push a directive to the local edge proxy to limit subsequent matching requests. So the next request is rejected without a round trip to the rate limiter.

Is the "local edge proxy" the kgateway? If so, I don't think there's anything within the proxy that can be configured to receive such data for routing decisions. If you're talking about a separate bespoke proxy, then external processing with "observability mode" is more appropriate. We don't currently expose that either, but it would be more suited for the use case of "out-of-band processing, push decision-making parameters to a separate proxy"

[davidxia]

have you considered using external processing for receiving the request data you care about (headers, body, trailers, all, some, etc)?

I was hoping to use as much open-source as possible and am looking at https://github.com/envoyproxy/ratelimit as the rate limiting service. Perhaps this doesn't support my use case and I need to implement something myself?

Is the "local edge proxy" the kgateway?

Yes, the local edge proxy here are the Envoy pods managed by kgateway.

Do you have any advice on how I should proceed? 🙏

[davidxia]

After reading the description for apply_on_stream_done again and again, I finally understand.

If true, the rate limit request will be applied when the stream completes...This is useful when the rate limit budget needs to reflect the response context that is not available on the request path.

This is orthogonal to what I want. Right now I only depend on information available in the request.

[davidxia]

@jbohanon after reading more, I created this separate feature request #14389. Does that one make more sense? If so, I can close this one. Thanks for your time.