Skip to content

Add filtering out of documents with error.message from latest indexes #15722

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add filtering out of documents with error.message from latest indexes #15722

wants to merge 2 commits into from
+142 −26

Conversation

@alexreal1314
Copy link
Contributor

@alexreal1314 alexreal1314 commented Oct 22, 2025 •
edited
Loading

Proposed commit message

Purpose of this PR is to filter out documents who contain error.message in the source indexes for all supported 3p integration and our native integration. This is in order to improve UI experience by filtering them out from the cdr workflows.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

  1. install one of the changed integrations - for example with wiz
  2. toggle Collect Wiz logs via API
  3. type in the wrong client id or client secret
  4. add an agent and let the integration ingest data
  5. vulnerability source index logs-wiz.vulnerability-default should contain a document with error.message field but no documents should be in the dest index security_solution-wiz.vulnerability_latest-v2.

Screenshots

source index - wiz:
image

dest index - wiz:
image

@alexreal1314 alexreal1314 force-pushed the 11031-latest-index-error-filtering branch from 005bfac to 4f4e661 Compare October 22, 2025 15:41
@alexreal1314 alexreal1314 changed the title add filtering out of documents with error.message from latest misconf… Add filtering out of documents with error.message from latest indexes Oct 22, 2025
@andrewkroh andrewkroh added Integration:google_scc Google Security Command Center Integration:wiz Wiz Integration:cloud_security_posture Security Posture Management Integration:rapid7_insightvm Rapid7 InsightVM Integration:m365_defender Microsoft Defender XDR Integration:qualys_vmdr Qualys VMDR Integration:aws AWS Integration:tenable_io Tenable Vulnerability Management Integration:microsoft_defender_endpoint Microsoft Defender for Endpoint labels Oct 22, 2025
@alexreal1314 alexreal1314 force-pushed the 11031-latest-index-error-filtering branch from 4f4e661 to 7bc623c Compare October 22, 2025 20:46
@alexreal1314 alexreal1314 added the Integration:microsoft_defender_cloud Microsoft Defender for Cloud label Oct 22, 2025
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Oct 22, 2025 •
edited
Loading

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@alexreal1314 alexreal1314 marked this pull request as ready for review October 23, 2025 06:01
@alexreal1314 alexreal1314 requested review from a team as code owners October 23, 2025 06:01
@alexreal1314 alexreal1314 self-assigned this Oct 23, 2025
@efd6
Copy link
Contributor

efd6 commented Oct 23, 2025

Is there an issue for this explaining why it's necessary?

@alexreal1314 alexreal1314 marked this pull request as draft October 23, 2025 06:45
@alexreal1314 alexreal1314 force-pushed the 11031-latest-index-error-filtering branch 2 times, most recently from 256f44a to a672def Compare October 23, 2025 07:57
@alexreal1314
Copy link
Contributor Author

Is there an issue for this explaining why it's necessary?

This PR closes this issue, the intention it to maintain clean latest indexes and prevent documents with errors reaching them.

@alexreal1314 alexreal1314 force-pushed the 11031-latest-index-error-filtering branch from 5fdc011 to b65571b Compare October 23, 2025 08:51
@alexreal1314 alexreal1314 added Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Cloud Security Cloud Security team [elastic/cloud-security-posture] labels Oct 23, 2025
@efd6
Copy link
Contributor

efd6 commented Oct 23, 2025

I think this should either be conditional on user configuration or done dynamically rather than my mutating the index.

@alexreal1314
Copy link
Contributor Author

I think this should either be conditional on user configuration or done dynamically rather than my mutating the index.

AFAIK we don't mutate any index, just filtering out findings containing errors messages.

@maxcold do we need to make the filtering configurable? didn't see such requirement.

@alexreal1314
add filtering out of documents with error.message from latest misconf…
ab22a4a 
…iguration and vulnerability index

change is added to all supported native and 3p integrations
@alexreal1314 alexreal1314 force-pushed the 11031-latest-index-error-filtering branch from b65571b to f8d6a64 Compare October 27, 2025 13:01
@maxcold
Copy link
Contributor

maxcold commented Oct 27, 2025

@alexreal1314 @efd6 I don't see a reason to make it configurable tbh. The sole reason the transforms exist today for these integrations is that we want the data from the latest transform destination index to be used in the Findings page in Security Solution. This change only affects the destination index, not a source index which is used by the users for their needs (dashbaords, alerts, whatever usercase pre tranform existence). Destination index is only for our internal use really and we don't want error documents to show up on the Findings page

@alexreal1314 alexreal1314 marked this pull request as ready for review October 27, 2025 16:32
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@andrewkroh andrewkroh added the Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] label Oct 27, 2025
@efd6
Copy link
Contributor

efd6 commented Oct 27, 2025

@maxcold That's reasonable.

efd6
efd6 reviewed Oct 27, 2025
View reviewed changes
packages/aws/changelog.yml Outdated
Comment on lines 12 to 14
- description: Update transform to filter out document containing an error.message from AWS Config, AWS Inspector, and AWS Security Hub latest indexes.
type: bugfix
link: https://github.com/elastic/integrations/pull/15722
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this here?

Copy link
Contributor Author

@alexreal1314 alexreal1314 Oct 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@efd6 could you elaborate please?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This text is an addition to a previous version that was added in #15737 and is a duplicate of the text above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@efd6 fixed, thanks.

@alexreal1314 alexreal1314 force-pushed the 11031-latest-index-error-filtering branch from f8d6a64 to 4b3069c Compare October 28, 2025 08:00
@alexreal1314 alexreal1314 requested a review from efd6 October 28, 2025 08:01
@elasticmachine
Copy link

💚 Build Succeeded

History

cc @alexreal1314

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maxcold maxcold maxcold approved these changes

@efd6 efd6 Awaiting requested review from efd6

At least 1 approving review is required to merge this pull request.

Assignees

@alexreal1314 alexreal1314

Labels

Integration:aws AWS Integration:cloud_security_posture Security Posture Management Integration:google_scc Google Security Command Center Integration:m365_defender Microsoft Defender XDR Integration:microsoft_defender_cloud Microsoft Defender for Cloud Integration:microsoft_defender_endpoint Microsoft Defender for Endpoint Integration:qualys_vmdr Qualys VMDR Integration:rapid7_insightvm Rapid7 InsightVM Integration:tenable_io Tenable Vulnerability Management Integration:wiz Wiz Team:Cloud Security Cloud Security team [elastic/cloud-security-posture] Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@alexreal1314 @efd6 @maxcold @elasticmachine @andrewkroh