Update codegen to automatically add chunked encoding signer property … #6551

zoewangg · 2025-11-06T23:03:56Z

Motivation and Context

When streaming operations have HTTP checksum traits, they need chunked encoding enabled in the auth signer. Previously, this had to be manually configured. This PR automates the process by detecting these operations during code generation and automatically adding the CHUNK_ENCODING_ENABLED property to their auth scheme metadata.

Additionally, the ModelAuthSchemeKnowledgeIndex class was refactored to improve maintainability and clarity.

Modifications

Automatic Chunked Encoding Detection:
• Added shouldUseChunkedEncoding() to detect streaming operations with HTTP checksum traits
• Automatically marks these operations with CHUNK_ENCODING_ENABLED property in auth scheme metadata
Refactored ModelAuthSchemeKnowledgeIndex:
• Extracted helper methods for better separation of concerns:
• operationsShouldUseChunkedEncoding() - Identifies operations needing chunked encoding
• processOperationsWithAuthTraits() - Handles main processing loop
• determineAuthScheme() - Decides between custom auth or service defaults
• processRemainingChunkedEncodingOperations() - Handles chunked ops with service defaults
• hasAuthTrait() - Filters operations and validates auth traits
• Generalized validation logic with hasCustomServiceAuthSchemeOverride() instead of hardcoded service checks
• Added javadocs explaining flow and special cases
Test Updates:
• Added test models for S3 and S3Control with chunked encoding scenarios
• Updated existing tests to verify chunked encoding property is correctly added

Testing

• Added new test cases for streaming operations with HTTP checksums
• Verified chunked encoding property is correctly added to auth metadata
• All existing unit tests pass
• Local mvn install succeeds

Types of changes

• [x] New feature (non-breaking change which adds functionality)

Checklist

• [x] I have read the CONTRIBUTING document
• [x] Local run of mvn install succeeds
• [x] My code follows the code style of this project
• [x] My change requires a change to the Javadoc documentation
• [x] I have updated the Javadoc documentation accordingly
• [x] I have added tests to cover my changes
• [x] All new and existing tests passed
• [ ] I have added a changelog entry
• [ ] My change is to implement 1.11 parity feature - N/A

License

• [x] I confirm that this pull request can be released under the Apache 2 license

zoewangg · 2025-11-06T23:46:52Z

...n/src/test/resources/software/amazon/awssdk/codegen/poet/client/c2j/rest-json/service-2.json

        "requestValidationModeMember": "ChecksumMode",
-        "responseAlgorithms": ["CRC32C", "CRC32", "SHA1", "SHA256"]
+        "responseAlgorithms": ["CRC32C", "CRC32", "SHA1", "SHA256"],
+        "requestAlgorithmMember": "ChecksumAlgorithm"


Added requestAlgorithmMember because per smithy doc https://smithy.io/2.0/aws/aws-core.html#aws-protocols-httpchecksum-trait, either requestAlgorithmMember or requestChecksumRequired is required

The httpChecksum trait MUST define at least one of the request checksumming behavior, by setting the requestAlgorithmMember or requestChecksumRequired property,

zoewangg · 2025-11-06T23:48:16Z

codegen/src/test/java/software/amazon/awssdk/codegen/poet/auth/scheme/AuthSchemeSpecTest.java

+                    .modelProvider(ClientTestModels::serviceS3)
                    .classSpecProvider(ModelBasedAuthSchemeProviderSpec::new)
-                    .caseName("mini-s3")
+                    .caseName("s3-test")


I changed it to use real s3 model because why not 🤷🏼‍♀️ and it may help us catch S3 issues

zoewangg · 2025-11-06T23:48:30Z

codegen/src/test/java/software/amazon/awssdk/codegen/poet/auth/scheme/AuthSchemeSpecTest.java

+                    .caseName("s3-test")
+                    .outputFileSuffix("fallback-provider")
+                    .build(),
+            // S3 control


Added s3 control because it uses legacy signature version

zoewangg · 2025-11-06T23:51:53Z

...main/java/software/amazon/awssdk/codegen/poet/auth/scheme/ModelAuthSchemeKnowledgeIndex.java

+    }
+
+    /**
+     * Returns a map from a list of operations to the list of auth-types modeled for those operations. The {@link AuthTrait}


I tried my best to explain what this method does here because it's quite hard to understand, lmk if it's still not clear.

FWIW, I think the explanation is good.

sugmanue

Minor nits, overall looks good to me.

...main/java/software/amazon/awssdk/codegen/poet/auth/scheme/ModelAuthSchemeKnowledgeIndex.java

sugmanue · 2025-11-11T18:35:32Z

...main/java/software/amazon/awssdk/codegen/poet/auth/scheme/ModelAuthSchemeKnowledgeIndex.java

+    }
+
+    /**
+     * Returns a map from a list of operations to the list of auth-types modeled for those operations. The {@link AuthTrait}


FWIW, I think the explanation is good.

sugmanue · 2025-11-11T18:51:44Z

...es/software/amazon/awssdk/codegen/poet/auth/scheme/s3-test-auth-scheme-default-provider.java

+                    RegionSet regionSet = Optional.ofNullable(params.regionSet()).orElseGet(
+                        () -> Optional.ofNullable(sigv4aAuthScheme.signingRegionSet())
+                                      .filter(set -> !CollectionUtils.isNullOrEmpty(set)).map(RegionSet::create).orElse(null));


Unrelated to this change, but, let's say, this can be improved, two allocs just to write this in a single expression is not a good tradeoff, in my not-so-humble opinion.

Yeah, to keep this PR small, I will create a separate PR for this

joviegas · 2025-11-20T23:28:55Z

...sources/software/amazon/awssdk/codegen/poet/client/c2j/s3control-test/endpoint-rule-set.json

@@ -0,0 +1,3380 @@
+{
+    "version": "1.0",


can we please keep this file empty or just keep minimal how much ever required for codegen test

Same for other resource files.

I used the real s3 and s3 control models as of when the PR was created on purpose because I figured it would help us catch s3 changes. LMK if you have concerns

…for streaming operations with http checksum trait

zoewangg · 2025-11-25T23:02:14Z

DefaultS3AuthSchemeProvider diff

➜ ~ diff -s baseline_s3_auth_scheme_provider.java new_s3_auth_scheme_provider.java
Files baseline_s3_auth_scheme_provider.java and new_s3_auth_scheme_provider.java are identical

DefaultS3FallbackAuthSchemeProvider diff

➜ ~ diff -s baseline_fallback_s3_auth_scheme_provider.java new_fallback_s3_auth_scheme_provider.java
Files baseline_fallback_s3_auth_scheme_provider.java and new_fallback_s3_auth_scheme_provider.java are identical

DefaultS3ControlAuthSchemeProvider diff

➜ ~ diff -s baseline_s3control_auth_scheme_provider.java new_s3control_auth_scheme_provider.java
43c43,45

<                 .putSignerProperty(AwsV4HttpSigner.REGION_NAME, params.region().id()).build());
---
>                 .putSignerProperty(AwsV4HttpSigner.REGION_NAME, params.region().id())
>                 .putSignerProperty(AwsV4HttpSigner.DOUBLE_URL_ENCODE, false)
>                 .putSignerProperty(AwsV4HttpSigner.NORMALIZE_PATH, false).build());

…ies since they are handled in codegen now

zoewangg · 2025-11-25T23:08:54Z

...tware/amazon/awssdk/services/s3control/internal/interceptors/ConfigureSignerInterceptor.java

- * https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
- */
-@SdkInternalApi
-public final class ConfigureSignerInterceptor implements ExecutionInterceptor {


This is no longer needed because it's handled in generated auth scheme provider now. See DefaultS3ControlAuthSchemeProvider diff in #6551 (comment)

sonarqubecloud · 2025-11-25T23:52:01Z

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
87.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

dagnir · 2025-11-26T01:35:54Z

...main/java/software/amazon/awssdk/codegen/poet/auth/scheme/ModelAuthSchemeKnowledgeIndex.java

    }

+    /**
+     * Determines if an operation should be included based on whether it has auth traits.


minor: It's not clear what the operation should be included in. Could probably just remove that part.

dagnir · 2025-11-26T01:53:37Z

...main/java/software/amazon/awssdk/codegen/poet/auth/scheme/ModelAuthSchemeKnowledgeIndex.java

+        Map<List<String>, List<AuthSchemeCodegenMetadata>> result = new LinkedHashMap<>();
+        Set<String> processedChunkedOps = new HashSet<>();
+
+        processOperationsWithAuthTraits(chunkedEncodingOps, serviceDefaults, result, processedChunkedOps);


processedChunkedOps is a bit hard to follow; is there a more descriptive name we can use here? i.e. what does it mean for something to be an "unprocessed" chunked operation? It doesn't use service defaults?

zoewangg requested a review from a team as a code owner November 6, 2025 23:03

zoewangg marked this pull request as draft November 6, 2025 23:04

zoewangg commented Nov 6, 2025

View reviewed changes

zoewangg force-pushed the zoewang/chunkedEncoding branch from baff246 to 839f640 Compare November 7, 2025 00:45

sugmanue approved these changes Nov 11, 2025

View reviewed changes

sugmanue reviewed Nov 11, 2025

View reviewed changes

joviegas reviewed Nov 20, 2025

View reviewed changes

joviegas approved these changes Nov 25, 2025

View reviewed changes

zoewangg marked this pull request as ready for review November 25, 2025 21:31

zoewangg added 2 commits November 25, 2025 13:45

Update codegen to automatically add chunked encoding signer property …

6c89a73

…for streaming operations with http checksum trait

Address feedback

324d5cd

zoewangg added 2 commits November 25, 2025 15:04

Remove unncessary s3 control interceptor that enables signing propert…

b608ea3

…ies since they are handled in codegen now

Add changelog entry

81c108f

zoewangg force-pushed the zoewang/chunkedEncoding branch from 839f640 to 81c108f Compare November 25, 2025 23:07

zoewangg commented Nov 25, 2025

View reviewed changes

dagnir reviewed Nov 26, 2025

View reviewed changes

Update codegen to automatically add chunked encoding signer property … #6551

Are you sure you want to change the base?

Update codegen to automatically add chunked encoding signer property … #6551

Uh oh!

Conversation

zoewangg commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation and Context

Modifications

Testing

Types of changes

Checklist

License

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sugmanue left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zoewangg commented Nov 25, 2025

DefaultS3AuthSchemeProvider diff

DefaultS3FallbackAuthSchemeProvider diff

DefaultS3ControlAuthSchemeProvider diff

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Nov 25, 2025

Quality Gate passed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zoewangg commented Nov 6, 2025 •

edited

Loading