Skip to content

[Netskope] Add alerts_events_v2 data stream to fetch the data for alerts_v2 and events_v2 from a single queue #15697

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

[Netskope] Add alerts_events_v2 data stream to fetch the data for alerts_v2 and events_v2 from a single queue #15697

wants to merge 2 commits into from
+5,325 −204

Conversation

@moxarth-rathod
Copy link
Contributor

Proposed commit message

netskope: add alerts_events_v2 data stream to fetch the data for alerts_v2 and events_v2 from a single queue

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/netskope directory.
  • Run the following command to run tests.

elastic-package test -v

Additionally, the following cloud credentials are required to setup:

AWS:

  • AWS_DEFAULT_PROFILE
  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_SESSION_TOKEN.

Related issues

@moxarth-rathod moxarth-rathod self-assigned this Oct 20, 2025
@moxarth-rathod moxarth-rathod requested a review from a team as a code owner October 20, 2025 17:05
@moxarth-rathod moxarth-rathod added breaking change Integration:netskope Netskope Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Sit-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Oct 20, 2025
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@andrewkroh andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Oct 20, 2025
leandrojmp
leandrojmp reviewed Oct 21, 2025
View reviewed changes
packages/netskope/data_stream/alerts_events_v2/agent/stream/aws-s3.yml.hbs Outdated
bucket_arn: {{bucket_arn}}
{{/if}}
{{#if number_of_workers}}
number_of_workers: {{number_of_workers}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @moxarth-rathod

This needs to be outside #if collect_s3_logs and #if queue_url as the setting number_of_workers is applied to both ways of getting data from S3, polling and SQS, the setting max_number_of_messages is ignored on agents higher than 8.16+

For more context this was reported on #13179 and fixed on multiple integrations on #13350

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I've made the changes accordingly.

leandrojmp
leandrojmp reviewed Oct 21, 2025
View reviewed changes
packages/netskope/_dev/build/docs/README.md Outdated

**Note**: It is recommended to use the combined alerts_events_v2 data stream rather than configuring the individual events_v2 or alerts_v2 data stream. The alerts_events_v2 stream automatically directs logs to the appropriate individual data streams.

If the individual v2 data streams, events_v2 or alerts_v2, are used via SQS, it is necessary to implement event-based bucket segregation.
Copy link
Contributor

@leandrojmp leandrojmp Oct 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may change in the future, but according to current Netskope documentation, if you choose any event type and alerts, they will be streamed together.

The user may choose to stream only events or to stream only alerts, but when choosing alerts and any other event type, they will be streamed together.

@elasticmachine
Copy link

elasticmachine commented Oct 23, 2025 •
edited
Loading

💔 Build Failed

Failed CI Steps

History

cc @moxarth-rathod

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leandrojmp leandrojmp Awaiting requested review from leandrojmp

At least 1 approving review is required to merge this pull request.

Assignees

@moxarth-rathod moxarth-rathod

Labels

breaking change documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:netskope Netskope Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Sit-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Netskope]: Using Log Streaming mode does not support to get both Events and Alerts

4 participants

@moxarth-rathod @elasticmachine @leandrojmp @andrewkroh