Skip to content

Commit 7ee5baa

Browse files
msailesmsailes
authored
feat: SQS Large message handling (#55)
1 parent 583d187 commit 7ee5baa

File tree

13 files changed

+664
-13
lines changed

13 files changed

+664
-13
lines changed

‎.github/workflows/build.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ jobs:
2020
env:
2121
OS: ${{ matrix.os }}
2222
JAVA: ${{ matrix.java-version }}
23+
AWS_REGION: eu-west-1
2324
steps:
2425
- uses: actions/checkout@v2
2526
- name: Setup java

‎docs/content/dummy.md‎

Lines changed: 0 additions & 8 deletions
This file was deleted.

‎docs/content/utilities/sqs_large_message_handling.mdx‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
---
2+
title: SQS Large Message Handling
3+
description: Utility
4+
---
5+
6+
The large message handling utility handles SQS messages which have had their payloads
7+
offloaded to S3 due to them being larger than the SQS maximum.
8+
9+
The utility automatically retrieves messages which have been offloaded to S3 using the
10+
[amazon-sqs-java-extended-client-lib](https://github.com/awslabs/amazon-sqs-java-extended-client-lib)
11+
client library. Once the message payloads have been processed successful the
12+
utility can delete the message payloads from S3.
13+
14+
This utility is compatible with versions *1.1.0+* of amazon-sqs-java-extended-client-lib.</p>
15+
16+
```xml
17+
<dependency>
18+
<groupId>com.amazonaws</groupId>
19+
<artifactId>amazon-sqs-java-extended-client-lib</artifactId>
20+
<version>1.1.0</version>
21+
</dependency>
22+
```
23+
24+
## Install
25+
26+
To install this utility, add the following dependency to your project.
27+
28+
```xml
29+
<dependency>
30+
<groupId>software.amazon.lambda</groupId>
31+
<artifactId>powertools-sqs</artifactId>
32+
<version>0.1.0-beta</version>
33+
</dependency>
34+
```
35+
36+
And configure the aspectj-maven-plugin to compile-time weave (CTW) the
37+
aws-lambda-powertools-java aspects into your project. You may already have this
38+
plugin in your pom. In that case add the depenedency to the `aspectLibraries`
39+
section.
40+
41+
```xml
42+
<build>
43+
<plugins>
44+
...
45+
<plugin>
46+
<groupId>org.codehaus.mojo</groupId>
47+
<artifactId>aspectj-maven-plugin</artifactId>
48+
<version>1.11</version>
49+
<configuration>
50+
<source>1.8</source>
51+
<target>1.8</target>
52+
<complianceLevel>1.8</complianceLevel>
53+
<aspectLibraries>
54+
...
55+
<aspectLibrary>
56+
<groupId>software.amazon.lambda</groupId>
57+
<artifactId>powertools-sqs</artifactId>
58+
</aspectLibrary>
59+
...
60+
</aspectLibraries>
61+
</configuration>
62+
<executions>
63+
<execution>
64+
<goals>
65+
<goal>compile</goal>
66+
</goals>
67+
</execution>
68+
</executions>
69+
</plugin>
70+
...
71+
</plugins>
72+
</build>
73+
```
74+
75+
The annotation `@LargeMessageHandler` should be used with the handleRequest method of a class
76+
which implements `com.amazonaws.services.lambda.runtime.RequestHandler` with
77+
`com.amazonaws.services.lambda.runtime.events.SQSEvent` as the first parameter.
78+
79+
```java
80+
public class SqsMessageHandler implements RequestHandler<SQSEvent, String> {
81+
82+
@Override
83+
@LargeMessageHandler
84+
public String handleRequest(SQSEvent sqsEvent, Context context) {
85+
// process messages
86+
87+
return "ok";
88+
}
89+
```
90+
91+
`@LargeMessageHandler` creates a default S3 Client `AmazonS3 amazonS3 = AmazonS3ClientBuilder.defaultClient()`.
92+
When the Lambda function is invoked with an event from SQS, each record received
93+
in the SQSEvent will be checked to see if it's body contains a payload which has
94+
been offloaded to S3. If it does then `getObject(bucket, key)` will be called,
95+
and the payload retrieved. If there is an error during this process then the
96+
function will fail with a `FailedProcessingLargePayloadException` exception.
97+
98+
If the request handler method returns without error then each payload will be
99+
deleted from S3 using `deleteObject(bucket, key)`
100+
101+
To disable deletion of payloads setting the following annotation parameter:
102+
103+
```java
104+
@LargeMessageHandler(deletePayloads=false)
105+
```

‎docs/gatsby-config.js‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,10 @@ module.exports = {
2525
'Core utilities': [
2626
'core/logging',
2727
'core/tracing'
28-
]
28+
],
29+
'Utilities': [
30+
'utilities/sqs_large_message_handling'
31+
],
2932
},
3033
navConfig: {
3134
'Serverless Best Practices video': {

‎example/HelloWorldFunction/pom.xml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,12 @@
1616
<dependency>
1717
<groupId>software.amazon.lambda</groupId>
1818
<artifactId>powertools-tracing</artifactId>
19-
<version>0.1.0-SNAPSHOT</version>
19+
<version>0.1.0-beta</version>
2020
</dependency>
2121
<dependency>
2222
<groupId>software.amazon.lambda</groupId>
2323
<artifactId>powertools-logging</artifactId>
24-
<version>0.1.0-SNAPSHOT</version>
24+
<version>0.1.0-beta</version>
2525
</dependency>
2626
<dependency>
2727
<groupId>com.amazonaws</groupId>

‎pom.xml‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@
3030
<module>powertools-core</module>
3131
<module>powertools-logging</module>
3232
<module>powertools-tracing</module>
33+
<module>powertools-sqs</module>
3334
</modules>
3435

3536
<scm>
@@ -50,9 +51,12 @@
5051
<log4j.version>2.13.3</log4j.version>
5152
<jackson.version>2.11.0</jackson.version>
5253
<aspectj.version>1.9.6</aspectj.version>
54+
<aws.sdk.version>2.14.4</aws.sdk.version>
5355
<aws.xray.recorder.version>2.4.0</aws.xray.recorder.version>
56+
<payloadoffloading-common.version>1.0.0</payloadoffloading-common.version>
5457
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
5558
<lambda.core.version>1.2.1</lambda.core.version>
59+
<lambda.events.version>3.1.0</lambda.events.version>
5660
<maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>
5761
<aspectj-maven-plugin.version>1.12.1</aspectj-maven-plugin.version>
5862
<maven-surefire-plugin.version>2.22.2</maven-surefire-plugin.version>
@@ -62,6 +66,7 @@
6266
<maven-javadoc-plugin.version>2.10.3</maven-javadoc-plugin.version>
6367
<maven-source-plugin.version>2.4</maven-source-plugin.version>
6468
<maven-gpg-plugin.version>1.6</maven-gpg-plugin.version>
69+
<junit-jupiter.version>5.6.2</junit-jupiter.version>
6570
</properties>
6671

6772
<distributionManagement>
@@ -83,6 +88,16 @@
8388
<artifactId>aws-lambda-java-core</artifactId>
8489
<version>${lambda.core.version}</version>
8590
</dependency>
91+
<dependency>
92+
<groupId>com.amazonaws</groupId>
93+
<artifactId>aws-lambda-java-events</artifactId>
94+
<version>${lambda.events.version}</version>
95+
</dependency>
96+
<dependency>
97+
<groupId>software.amazon.payloadoffloading</groupId>
98+
<artifactId>payloadoffloading-common</artifactId>
99+
<version>${payloadoffloading-common.version}</version>
100+
</dependency>
86101
<dependency>
87102
<groupId>org.aspectj</groupId>
88103
<artifactId>aspectjrt</artifactId>
@@ -133,13 +148,19 @@
133148
<dependency>
134149
<groupId>org.junit.jupiter</groupId>
135150
<artifactId>junit-jupiter-api</artifactId>
136-
<version>5.6.2</version>
151+
<version>${junit-jupiter.version}</version>
137152
<scope>test</scope>
138153
</dependency>
139154
<dependency>
140155
<groupId>org.junit.jupiter</groupId>
141156
<artifactId>junit-jupiter-engine</artifactId>
142-
<version>5.6.2</version>
157+
<version>${junit-jupiter.version}</version>
158+
<scope>test</scope>
159+
</dependency>
160+
<dependency>
161+
<groupId>org.junit.jupiter</groupId>
162+
<artifactId>junit-jupiter-params</artifactId>
163+
<version>${junit-jupiter.version}</version>
143164
<scope>test</scope>
144165
</dependency>
145166
<dependency>

‎powertools-sqs/pom.xml‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0"
3+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
4+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
5+
<modelVersion>4.0.0</modelVersion>
6+
7+
<artifactId>powertools-sqs</artifactId>
8+
<packaging>jar</packaging>
9+
10+
<parent>
11+
<artifactId>powertools-parent</artifactId>
12+
<groupId>software.amazon.lambda</groupId>
13+
<version>0.1.0-beta</version>
14+
</parent>
15+
16+
<name>AWS Lambda Powertools Java library SQS</name>
17+
<description>
18+
A suite of utilities for AWS Lambda Functions that makes tracing with AWS X-Ray, structured logging and creating custom metrics asynchronously easier.
19+
</description>
20+
<url>https://aws.amazon.com/lambda/</url>
21+
<issueManagement>
22+
<system>GitHub Issues</system>
23+
<url>https://github.com/awslabs/aws-lambda-powertools-java/issues</url>
24+
</issueManagement>
25+
<scm>
26+
<url>https://github.com/awslabs/aws-lambda-powertools-java.git</url>
27+
</scm>
28+
<developers>
29+
<developer>
30+
<name>AWS Lambda Powertools team</name>
31+
<organization>Amazon Web Services</organization>
32+
<organizationUrl>https://aws.amazon.com/</organizationUrl>
33+
</developer>
34+
</developers>
35+
36+
<distributionManagement>
37+
<snapshotRepository>
38+
<id>ossrh</id>
39+
<url>https://aws.oss.sonatype.org/content/repositories/snapshots</url>
40+
</snapshotRepository>
41+
</distributionManagement>
42+
43+
<dependencies>
44+
<dependency>
45+
<groupId>software.amazon.lambda</groupId>
46+
<artifactId>powertools-core</artifactId>
47+
</dependency>
48+
<dependency>
49+
<groupId>com.amazonaws</groupId>
50+
<artifactId>aws-lambda-java-core</artifactId>
51+
</dependency>
52+
<dependency>
53+
<groupId>com.amazonaws</groupId>
54+
<artifactId>aws-lambda-java-events</artifactId>
55+
</dependency>
56+
<dependency>
57+
<groupId>software.amazon.payloadoffloading</groupId>
58+
<artifactId>payloadoffloading-common</artifactId>
59+
</dependency>
60+
61+
<dependency>
62+
<groupId>org.aspectj</groupId>
63+
<artifactId>aspectjrt</artifactId>
64+
</dependency>
65+
66+
<!-- Test dependencies -->
67+
<dependency>
68+
<groupId>org.junit.jupiter</groupId>
69+
<artifactId>junit-jupiter-api</artifactId>
70+
<scope>test</scope>
71+
</dependency>
72+
<dependency>
73+
<groupId>org.junit.jupiter</groupId>
74+
<artifactId>junit-jupiter-engine</artifactId>
75+
<scope>test</scope>
76+
</dependency>
77+
<dependency>
78+
<groupId>org.junit.jupiter</groupId>
79+
<artifactId>junit-jupiter-params</artifactId>
80+
<scope>test</scope>
81+
</dependency>
82+
<dependency>
83+
<groupId>org.apache.commons</groupId>
84+
<artifactId>commons-lang3</artifactId>
85+
<scope>test</scope>
86+
</dependency>
87+
<dependency>
88+
<groupId>org.mockito</groupId>
89+
<artifactId>mockito-core</artifactId>
90+
<scope>test</scope>
91+
</dependency>
92+
<dependency>
93+
<groupId>org.aspectj</groupId>
94+
<artifactId>aspectjweaver</artifactId>
95+
<scope>test</scope>
96+
</dependency>
97+
<dependency>
98+
<groupId>org.assertj</groupId>
99+
<artifactId>assertj-core</artifactId>
100+
<scope>test</scope>
101+
</dependency>
102+
</dependencies>
103+
104+
</project>

‎powertools-sqs/src/main/java/software/amazon/lambda/powertools/sqs/LargeMessageHandler.java‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
package software.amazon.lambda.powertools.sqs;
2+
3+
import java.lang.annotation.ElementType;
4+
import java.lang.annotation.Retention;
5+
import java.lang.annotation.RetentionPolicy;
6+
import java.lang.annotation.Target;
7+
8+
/**
9+
* {@code LargeMessageHandler} is used to signal that the annotated method
10+
* should be extended to handle large SQS messages which have been offloaded
11+
* to S3
12+
*
13+
* <p>{@code LargeMessageHandler} automatically retrieves and deletes messages
14+
* which have been offloaded to S3 using the {@code amazon-sqs-java-extended-client-lib}
15+
* client library.</p>
16+
*
17+
* <p>This version of the {@code LargeMessageHandler} is compatible with version
18+
* 1.1.0+ of {@code amazon-sqs-java-extended-client-lib}.</p>
19+
*
20+
* <pre>
21+
* &lt;dependency&gt;
22+
* &lt;groupId&gt;com.amazonaws&lt;/groupId&gt;
23+
* &lt;artifactId&gt;amazon-sqs-java-extended-client-lib&lt;/artifactId&gt;
24+
* &lt;version&gt;1.1.0&lt;/version&gt;
25+
* &lt;/dependency&gt;
26+
* </pre>
27+
*
28+
* <p>{@code LargeMessageHandler} should be used with the handleRequest method of a class
29+
* which implements {@code com.amazonaws.services.lambda.runtime.RequestHandler} with
30+
* {@code com.amazonaws.services.lambda.runtime.events.SQSEvent} as the first parameter.</p>
31+
*
32+
* <pre>
33+
* public class SqsMessageHandler implements RequestHandler<SQSEvent, String> {
34+
*
35+
* {@literal @}Override
36+
* {@literal @}LargeMessageHandler
37+
* public String handleRequest(SQSEvent sqsEvent, Context context) {
38+
*
39+
* // process messages
40+
*
41+
* return "ok";
42+
* }
43+
*
44+
* ...
45+
* </pre>
46+
*
47+
* <p>Using the default S3 Client {@code AmazonS3 amazonS3 = AmazonS3ClientBuilder.defaultClient();}
48+
* each record received in the SQSEvent {@code LargeMessageHandler} will checked
49+
* to see if it's body contains a payload which has been offloaded to S3. If it
50+
* does then {@code getObject(bucket, key)} will be called and the payload
51+
* retrieved.</p>
52+
*
53+
* <p><b>Note</b>: Retreiving payloads from S3 will increase the duration of the
54+
* Lambda function.</p>
55+
*
56+
* <p>If the request handler method returns then each payload will be deleted
57+
* from S3 using {@code deleteObject(bucket, key)}</p>
58+
*
59+
* <p>To disable deletion of payloads setting the following annotation parameter
60+
* {@code @LargeMessageHandler(deletePayloads=false)}</p>
61+
*
62+
*/
63+
@Retention(RetentionPolicy.RUNTIME)
64+
@Target(ElementType.METHOD)
65+
public @interface LargeMessageHandler {
66+
67+
boolean deletePayloads() default true;
68+
}

0 commit comments

Comments
 (0)