feat: project setup and upload file #1712

0marperez · 2025-10-17T13:55:48Z

Issue #

N/A

Description of changes

Project setup
Publishing config
Transfer manager client
Business metric
Transfer interceptors
Upload file operation
- Concurrent uploads
- MPU part buffering
- Code generated IO
- Code generated type converters

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

github-actions · 2025-10-17T14:02:55Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T14:14:53Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T14:42:33Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T14:53:02Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T15:11:43Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

lauzadis

Nice start

...mmon/src/aws/sdk/kotlin/runtime/http/interceptors/businessmetrics/AwsBusinessMetricsUtils.kt

hll/build.gradle.kts

hll/s3-transfer-manager/build.gradle.kts

hll/s3-transfer-manager/jvm/test/aws/sdk/kotlin/hll/s3transfermanager/UploadFileTest.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/utils/UploadFile.kt

...ransfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/BusinessMetricInterceptor.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

...-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/model/UploadFileRequest.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/utils/UploadFile.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/TransferInterceptor.kt

hll/s3-transfer-manager/build.gradle.kts

github-actions · 2025-10-21T14:07:26Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-30T03:03:52Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

github-actions · 2025-10-30T04:09:45Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

github-actions · 2025-10-30T15:51:50Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

github-actions · 2025-10-30T16:13:52Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

gradle/libs.versions.toml

...n/kotlin/aws/sdk/kotlin/hll/dynamodbmapper/codegen/operations/rendering/OperationRenderer.kt

hll/s3-transfer-manager-codegen/build.gradle.kts

...odegen/src/main/kotlin/aws/sdk/kotlin/hll/s3transfermanager/codegen/mappings/MappingTypes.kt

hll/s3-transfer-manager/build.gradle.kts

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

...-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/UploadFile.kt

lauzadis · 2025-10-30T17:45:32Z

...ger/common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/HelperFunctions.kt

+    val targetNumberOfParts = contentLength / targetPartSize
+    return if (targetNumberOfParts > MAX_NUMBER_PARTS) {
+        ceilDiv(contentLength, MAX_NUMBER_PARTS).also {
+            logger.warn { "Target part size is too small to meet the $MAX_NUMBER_PARTS S3 part limit. Increasing part size to $it" }


Did we clarify with the spec author what level this should be logged at?

There's nothing in the spec mentioning logging a message when the configured part size isn't used btw.

The spec author uses DEBUG

Personally I like INFO level for this. The KDocs already indicate that the value is a target not a guarantee.

That said, I think we should make the message more descriptive. Ideally we'd say something like:

The target part size of <configured-value> bytes is too small upload <object-name> in <max-num-parts> parts (the maximum allowed by S3). The object will be uploaded in parts of <calculated-part-size> bytes instead.

...mon/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/hooks/InitiateTransfer.kt

...ger/common/test/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/UploadFileTest.kt

github-actions · 2025-10-30T20:40:09Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

hll/s3-transfer-manager-codegen/build.gradle.kts

ianbotsf · 2025-10-31T22:54:27Z

settings.gradle.kts

+if ("s3".isBootstrappedService) {
+    include(":hll:s3-transfer-manager")
+    include(":hll:s3-transfer-manager-codegen")
+} else {
+    logger.warn(":services:s3 is not bootstrapped, skipping :hll:s3-transfer-manager and subprojects")
+}


Comment: Excellent!

ianbotsf · 2025-10-31T22:56:59Z

gradle/libs.versions.toml

+smithy-kotlin-http-test-jvm = { module = "aws.smithy.kotlin:http-test-jvm", version.ref = "smithy-kotlin-runtime-version" }
+smithy-kotlin-testing-jvm = { module = "aws.smithy.kotlin:testing-jvm", version.ref = "smithy-kotlin-runtime-version" }


Question: Why are we explicitly depending on JVM target packages? Generally we rely on the common KMP packages (e.g., http-test, testing) because Gradle's Kotlin plugin is supposed to handle target resolution.

ianbotsf · 2025-10-31T22:58:52Z

hll/build.gradle.kts

+
+    "s3-transfer-manager-codegen", // TODO: Disable publishing ?


Comment: 👍 Yes, this is in the right place. We don't want to publish this since we have no use case for it right now. We can scratch the TODO.

This doesn't disable publication right? Just API validation and docgen

ianbotsf · 2025-10-31T22:59:45Z

hll/hll-codegen/build.gradle.kts

+
+java {
+    sourceCompatibility = JavaVersion.VERSION_1_8
+    targetCompatibility = JavaVersion.VERSION_1_8
+}
+
+tasks.withType<KotlinCompile> {
+    compilerOptions {
+        jvmTarget.set(JvmTarget.JVM_1_8)
+    }
+}


Question: Why are these additions necessary?

ianbotsf · 2025-11-03T18:16:46Z

...-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/UploadFile.kt

+    val mpuUploadId = initiateTransfer(
+        multipartUpload,
+        transferContext,
+        contentLength,
+        uploadFileRequest,
+        interceptors,
+        client,
+    )
+
+    val uploadedParts = transferBytes(
+        multipartUpload,
+        contentLength,
+        partSizeBytes,
+        logger,
+        uploadFileRequest,
+        transferContext,
+        mpuUploadId,
+        interceptors,
+        client,
+        maxInMemoryParts,
+        maxConcurrentPartUploads,
+    )
+
+    completeTransfer(
+        multipartUpload,
+        transferContext,
+        uploadFileRequest,
+        mpuUploadId,
+        uploadedParts,
+        interceptors,
+        client,
+    )


Style: The volume of arguments passed into these functions is too great. Can any of these be grouped into objects, derived from other parameters, etc.? This level of data coupling is an indicator that we might be better served modelling a base operation type which can be implemented for each operation type (e.g., UploadFile) or for each subtype (e.g., UploadFileSingle and UploadFileMultipart), which would reduce the amount of if (multipartUpload) calls.

ianbotsf · 2025-11-03T18:23:20Z

...c/main/kotlin/aws/sdk/kotlin/hll/s3transfermanager/codegen/mappings/uploadfile/Converters.kt

+            "tagging",
+            "websiteRedirectLocation",
+        ),
+        additionalLogic = "contentLength = [email protected]?.contentLength",


Nit: ByteStream's contentLength is nullable because it's not always possible to derive the length of a byte stream automatically. That's why S3's PutObjectRequest has a separate contentLength field users can populate with knowledge the SDK doesn't necessarily have. I think we need contentLength to be user-configurable and to prefer that value when it's set (otherwise, falling back to body?.contentLength).

ianbotsf · 2025-11-03T18:28:23Z

...common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/hooks/TransferBytes.kt

+/**
+ * Represents a part in a multipart upload.
+ *
+ * @param number The part number.
+ * @param bytes The bytes of the part.
+ */
+internal data class Part(
+    val number: Int,
+    val bytes: SdkBuffer,
+)


Nit: Could be private.

ianbotsf · 2025-11-03T18:32:36Z

...common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/hooks/TransferBytes.kt

+) = produce(
+    capacity = maxInMemoryParts,
+) {


Correctness: Isn't maxInMemoryParts supposed to limit the parts for the entire S3TM? This looks like it applies to individual objects but we'll be parallelizing multi-object transfers.

ianbotsf · 2025-11-03T18:36:07Z

...common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/hooks/TransferBytes.kt

+        } catch (uploadPartException: Exception) {
+            try {
+                client.abortMultipartUpload {
+                    bucket = uploadFileRequest.bucket
+                    expectedBucketOwner = uploadFileRequest.expectedBucketOwner
+                    key = uploadFileRequest.key
+                    requestPayer = uploadFileRequest.requestPayer
+                    uploadId = mpuUploadId
+                }
+                throw S3TransferManagerException("Multipart upload failed (ID: $mpuUploadId). One or more parts could not be uploaded", uploadPartException)
+            } catch (abortException: Exception) {
+                throw S3TransferManagerException("Multipart upload failed (ID: $mpuUploadId). Unable to abort multipart upload.", abortException)
+                    .also { it.addSuppressed(uploadPartException) }
+            }
+        }


Nit: This should probably log a WARN before aborting the multipart upload.

lauzadis · 2025-11-03T19:13:26Z

hll/build.gradle.kts

+
+    "s3-transfer-manager-codegen", // TODO: Disable publishing ?


This doesn't disable publication right? Just API validation and docgen

lauzadis · 2025-11-03T19:19:16Z

...c/main/kotlin/aws/sdk/kotlin/hll/s3transfermanager/codegen/mappings/uploadfile/Converters.kt

+internal val uploadFileConversions = listOf(
+    ConversionMapping(
+        source = TypeRef(
+            "aws.sdk.kotlin.services.s3.model",
+            "PutObjectResponse",
+        ),
+        destination = TypeRef(
+            "aws.sdk.kotlin.hll.s3transfermanager.model",
+            "UploadFileResponse",
+        ),
+        setOf(
+            "bucketKeyEnabled",
+            "checksumCrc32",
+            "checksumCrc32C",
+            "checksumCrc64Nvme",
+            "checksumSha1",
+            "checksumSha256",
+            "checksumType",
+            "eTag",
+            "expiration",


I don't think a data string will make this any more readable / maintainable? The internal spec already has this modeled as a JSON list, we can make changes (which I believe are unlikely) by inspecting the diff of that file

lauzadis · 2025-11-03T19:23:10Z

hll/s3-transfer-manager/build.gradle.kts

+        commonTest {
+            dependencies {
+                implementation(libs.smithy.kotlin.http.test.jvm)
+                implementation(libs.smithy.kotlin.testing.jvm)
+            }
+        }


Ian already commented on this, but JVM-only dependencies in commonTest won't work

lauzadis · 2025-11-03T19:27:25Z

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

+    /**
+     * Preferred part size for multipart uploads.
+     * If using this size would require more than 10,000 parts (the S3 limit),
+     * the smallest possible part size that results in 10,000 parts is used instead.
+     *
+     * Default to 8,000,000 bytes.
+     */
+    public val partSizeBytes: Long = builder.partSizeBytes


This is my fault since I left a comment asking to change it to just partSize to simplify the name. We are logging a warning when deviating from the configured part size. It's not a strong opinion so I will let @0marperez make the decision

#1712

0marperez added 6 commits October 6, 2025 17:39

feat: s3 transfer manager client

bc2fa0a

saving work - upload file

6895fc3

self review

c9486f1

pull from main

22243e7

misc: merge from main

dbff057

finish upload file

835866a

provide static dummy credentials to tests

8edf604

0marperez added the no-changelog Indicates that a changelog entry isn't required for a pull request. Use sparingly. label Oct 17, 2025

add s3 as a test dependency?

4137873

refactor tests to be JVM only

d84516d

Depend on missing 'DefaultAwsSigner'

9d5ec07

0marperez marked this pull request as ready for review October 17, 2025 15:14

0marperez requested a review from a team as a code owner October 17, 2025 15:14

lauzadis requested changes Oct 17, 2025

View reviewed changes

Merge branch 'main' of github.com:aws/aws-sdk-kotlin into s3-tm-dev

71ea5b7

ianbotsf requested changes Oct 20, 2025

View reviewed changes

feedback

fb2417f

0marperez mentioned this pull request Oct 30, 2025

feat: sdk source readRemaining smithy-lang/smithy-kotlin#1454

Open

feedback v2

a193a2f

add missing KDocs and self review

010bd4b

lauzadis mentioned this pull request Oct 30, 2025

S3 Transfer Manager #972

Open

1 task

remove old TODO

8a821d6

latest feedback

ac6c88e

lauzadis reviewed Oct 30, 2025

View reviewed changes

feedback v3

32aae24

ianbotsf requested changes Nov 3, 2025

View reviewed changes

lauzadis approved these changes Nov 3, 2025

View reviewed changes

		smithy-kotlin-http-test-jvm = { module = "aws.smithy.kotlin:http-test-jvm", version.ref = "smithy-kotlin-runtime-version" }
		smithy-kotlin-testing-jvm = { module = "aws.smithy.kotlin:testing-jvm", version.ref = "smithy-kotlin-runtime-version" }


		"s3-transfer-manager-codegen", // TODO: Disable publishing ?

feat: project setup and upload file #1712

Are you sure you want to change the base?

feat: project setup and upload file #1712

Uh oh!

Conversation

0marperez commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue #

Description of changes

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

lauzadis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 21, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0marperez Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

0marperez commented Oct 17, 2025 •

edited

Loading

0marperez Oct 30, 2025 •

edited

Loading