Skip to content

Conversation

@efd6
Copy link
Contributor

@efd6 efd6 commented Oct 24, 2025

The awss3 input incorrectly complains about the absence of a Records field in SQS messages. Valid SQS messages are possible without a Records field as described in the AWS documentation[1].

[1]https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html#notification-content-structure-examples

See also elastic/integrations#15472 (comment).

…by awss3 input

The awss3 input incorrectly complains about the absence of a Records
field in SQS messages. Valid SQS messages are possible without a Records
field as described in the AWS documentation[1].

[1]https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html#notification-content-structure-examples
@efd6 efd6 self-assigned this Oct 24, 2025
@efd6 efd6 added the enhancement New feature or request label Oct 24, 2025
@efd6 efd6 requested a review from a team October 24, 2025 05:53
@efd6 efd6 marked this pull request as ready for review October 24, 2025 05:57
@efd6
Copy link
Contributor Author

efd6 commented Oct 26, 2025

/test

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

cc @efd6

Comment on lines +159 to +161
// This is excluded because the awss3 input incorrectly complains about missing Records fields in s3:TestEvent messages.
// See https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html#notification-content-structure-examples.
regexp.MustCompile(`state changed .* \(HEALTHY->DEGRADED\): .* the message is an invalid S3 notification: missing Records field`),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One doubt here, is this exception required even if applying one of the suggestions mentioned in elastic/integrations#15472 (comment) ?

It looks like that if one of those suggestions mentioned in the integrations issue is applied, this exception/exclude would not be needed:

  • with the first suggestion, it would imply that there is no error anymore
  • with the second suggestion, IIUC the error reported in Elastic Agent logs would be a valid error since it won't be an s3:TestEvent message.

Would it be needed to wait to merge this PR depending on the action taken in the integrations issue ?
cc @jsoriano

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a problem with adding this exception. Even if we apply the changes mentioned in elastic/integrations#15472 (comment), it will be useful for versions of the stack without these fixes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The situation here is not ideal, but this is IMO the least worst choice. We can always remove this exception in the future when sufficiently many awss3 input integrations have been migrated to the fixed version of the input.

Comment on lines +159 to +161
// This is excluded because the awss3 input incorrectly complains about missing Records fields in s3:TestEvent messages.
// See https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html#notification-content-structure-examples.
regexp.MustCompile(`state changed .* \(HEALTHY->DEGRADED\): .* the message is an invalid S3 notification: missing Records field`),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a problem with adding this exception. Even if we apply the changes mentioned in elastic/integrations#15472 (comment), it will be useful for versions of the stack without these fixes.

@efd6 efd6 merged commit 01dc974 into elastic:main Oct 27, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants