Skip to content

💥 Payload limit enforcement#1363

Open
jmaeagle99 wants to merge 1 commit into
mainfrom
payload-limits
Open

💥 Payload limit enforcement#1363
jmaeagle99 wants to merge 1 commit into
mainfrom
payload-limits

Conversation

@jmaeagle99

@jmaeagle99 jmaeagle99 commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

What was changed

Builds on the existing payload-field classification codegen to add end-to-end payload/memo
size-limit enforcement in the client and worker.

  • Validate every outbound request via generated dispatcher; error-level violation fails with an invalid argument status carrying the PayloadLimitViolation as its source.
  • Warning thresholds configured on ConnectionOptions as payload_limits PayloadLimitsOptions field. Default is to warn payloads at 512 KiB and memos at 2 KiB, same defaults as server.
  • Error limits are read by workers when describing the namespace and set on the PayloadLimitsClient decorator that wraps the existing connection decorator. Each request has the error limits (if set) attached as a request extension.
  • On error limit violation, worker converts requests to their corresponding failures and the operations are retryable:
    • WFT completion → RespondWorkflowTaskFailed
    • Activity completion / cancel → RespondActivityTaskFailed
    • Activity heartbeat → RespondActivityTaskFailed and report activity as cancelled
    • Nexus completion → RespondNexusTaskFailed
  • Logs and task-failure messages render the same TMPRL1103 string.
  • Richer completion-span context (namespace/workflow_id/type/attempt/worker_id) threaded through for log forwarding.
  • New FailureReason::PayloadsTooLarge metric reason.

Why?

Prevent server from hard-failing workflows, activities, and operations due to payloads/memo size limit violations. Allows customers to potentially update their workflows, activities, and operations to reduce the oversized payloads/memos.

Checklist

  1. How was this tested: New unit tests and integration tests
  2. Any docs updates needed? Yes

@jmaeagle99 jmaeagle99 force-pushed the payload-limits branch 6 times, most recently from c42f207 to aebcc45 Compare July 1, 2026 02:46
@jmaeagle99 jmaeagle99 marked this pull request as ready for review July 1, 2026 05:20
@jmaeagle99 jmaeagle99 requested a review from a team as a code owner July 1, 2026 05:20

@jmaeagle99 jmaeagle99 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some self-review comments. If you have ideas/opinions, would love to hear them.

pub struct PayloadLimitsOptions {
/// Warning threshold (bytes) for the size of an outbound payload-bearing field; over-threshold
/// fields are logged but still sent to server. Defaults to 512 KiB. Set to `0` to disable.
pub payloads_size_warn: u64,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might flip the naming of these fields to:

  • payloads_warn_size
  • memo_warn_size
    I think it would allow better grouping of related settings together if we ever need to add more for each severity level e.g. payload_warn_fancy_feature and I think it reads better when saying them because there aren't two S's next to each other like there is in payloads_size_warn.

This suggestion is contrary to what we did for Python and Go. Also note that the payloads field is plural here where it is singular in Python and Go; I intentionally did that since the sizing is applied to payload-bearing fields, which include single payload fields, repeated payload fields (a sequence), and the Payloads proto message.

So options are:

  1. Leave as-is.
  2. Match Python and Go naming for consistency sake.
  3. Rename to payloads_warn_size and memo_warn_size for (IMO) maringally better naming and conceptualization.
  4. Some other better suggestion.

Naming is hard.

Comment thread crates/client/src/grpc.rs
F: Send + Sync + Unpin + 'static,
{
// Validate payload sizes after any request mutation but before encoding/metrics.
validate_request_payload_limits(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is situated right before the gRPC message is sent so that it is the last thing in the gRPC client chain that could modify payloads and memos. Things I need to check here are:

  • This needs to be checked before gRPC request metrics are emitted so that we aren't double counting attempts e.g. a WFT completion that is rejected due to payload sizes should not emit a gRPC request metric for that message.
  • I recall there is a grpc override callback. This probably should occur before that callback is invoked.

@jmaeagle99 jmaeagle99 changed the title feat: payload limit enforcement 💥 Payload limit enforcement Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant