💥 Payload limit enforcement#1363
Conversation
c42f207 to
aebcc45
Compare
aebcc45 to
a9f694c
Compare
jmaeagle99
left a comment
There was a problem hiding this comment.
Some self-review comments. If you have ideas/opinions, would love to hear them.
| pub struct PayloadLimitsOptions { | ||
| /// Warning threshold (bytes) for the size of an outbound payload-bearing field; over-threshold | ||
| /// fields are logged but still sent to server. Defaults to 512 KiB. Set to `0` to disable. | ||
| pub payloads_size_warn: u64, |
There was a problem hiding this comment.
I might flip the naming of these fields to:
payloads_warn_sizememo_warn_size
I think it would allow better grouping of related settings together if we ever need to add more for each severity level e.g.payload_warn_fancy_featureand I think it reads better when saying them because there aren't two S's next to each other like there is inpayloads_size_warn.
This suggestion is contrary to what we did for Python and Go. Also note that the payloads field is plural here where it is singular in Python and Go; I intentionally did that since the sizing is applied to payload-bearing fields, which include single payload fields, repeated payload fields (a sequence), and the Payloads proto message.
So options are:
- Leave as-is.
- Match Python and Go naming for consistency sake.
- Rename to
payloads_warn_sizeandmemo_warn_sizefor (IMO) maringally better naming and conceptualization. - Some other better suggestion.
Naming is hard.
| F: Send + Sync + Unpin + 'static, | ||
| { | ||
| // Validate payload sizes after any request mutation but before encoding/metrics. | ||
| validate_request_payload_limits( |
There was a problem hiding this comment.
This is situated right before the gRPC message is sent so that it is the last thing in the gRPC client chain that could modify payloads and memos. Things I need to check here are:
- This needs to be checked before gRPC request metrics are emitted so that we aren't double counting attempts e.g. a WFT completion that is rejected due to payload sizes should not emit a gRPC request metric for that message.
- I recall there is a grpc override callback. This probably should occur before that callback is invoked.
What was changed
Builds on the existing payload-field classification codegen to add end-to-end payload/memo
size-limit enforcement in the client and worker.
PayloadLimitViolationas its source.ConnectionOptionsaspayload_limits PayloadLimitsOptionsfield. Default is to warn payloads at 512 KiB and memos at 2 KiB, same defaults as server.PayloadLimitsClientdecorator that wraps the existing connection decorator. Each request has the error limits (if set) attached as a request extension.RespondWorkflowTaskFailedRespondActivityTaskFailedRespondActivityTaskFailedand report activity as cancelledRespondNexusTaskFailedFailureReason::PayloadsTooLargemetric reason.Why?
Prevent server from hard-failing workflows, activities, and operations due to payloads/memo size limit violations. Allows customers to potentially update their workflows, activities, and operations to reduce the oversized payloads/memos.
Checklist