Skip to content

fix: reduce no-match automation telemetry volume - #767

Merged
malhotra5 merged 2 commits into
mainfrom
reduce-no-match-automation-telemetry
Nov 17, 2026
Merged

malhotra5 merged 2 commits into
mainfrom
reduce-no-match-automation-telemetry

Conversation

@malhotra5

@malhotra5 malhotra5 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Stop sending automation_event_matched product telemetry to PostHog when an accepted webhook event matches zero automations.

The service still records the event in integration_events and still emits the internal observability span event with matched_count=0, so operational evidence remains available without paying PostHog volume for no-op webhook traffic.

This also renumbers the new webhook protocol Alembic migration from 038 to 039 because latest main already contains 038_add_run_deferred_until.py; the duplicate revision was blocking the PR unit-test job after rebasing onto main.

Why

The Agent Canvas PostHog spike starting Nov 12 was dominated by no-match GitHub workflow-run deliveries. Most of these events had matched_count == 0, so capturing every no-op match check created high product analytics volume without representing user-visible automation work.

Tests

  • uv run pytest tests/test_ingest.py -k 'telemetry or product_telemetry'
  • uv run pytest tests/test_migration_history.py::test_migrations_do_not_reuse_published_revision_ids
  • uv run pre-commit run --files openhands/automation/ingest.py tests/test_ingest.py migrations/versions/039_add_webhook_protocol_options.py

This PR was created by an AI agent (OpenHands) on behalf of the user.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Warning

Your comment is too long (maximum is 65536 characters), so the coverage report was not added. See the job log for how to reduce it.

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Reviewed the diff on head b27b878 against main, the surrounding ingest/telemetry/trigger-matcher code, and the test suite.

Assessment

  • The if matched: gate around capture_automation_event("automation_event_matched", ...) only suppresses the PostHog capture when zero automations matched. The integration_events row (record.matched_count) and the internal observability span event (automation.event.matched with automation.event.matched_count) are both emitted before the gate, so operational evidence for no-match deliveries is retained as described.
  • No lost product telemetry beyond the intended event: automation_run_created / automation_conversation_continued only fire inside the matched loop, and grep confirms no other emitter or consumer of automation_event_matched.
  • The new test deterministically drives the no-match path (on: "message" vs event key app_mention per matches_trigger semantics) and asserts real behavior end-to-end through accept_event with the telemetry HTTP client captured, including the persisted matched_count == 0 row.

Verification

  • unit-tests, backend, windows-tests, and all image/description checks pass on this head. Local lint and type checks (ruff format/check, pycodestyle, pyright) pass on both changed files. DB-backed tests require a PostgreSQL testcontainer that cannot run in this review sandbox, so suite correctness is verified via the green unit-tests CI run on the exact head plus local static verification.
  • Non-blocking note: the pr-title / Lint PR title (conventional) check fails because the title lacks a Conventional Commits prefix (e.g. feat:/chore:). This affects release-notes label grouping only; consider a retitle before merge.

No material correctness, security, or design findings.

✅ APPROVED

Co-authored-by: openhands <openhands@all-hands.dev>
@malhotra5
malhotra5 force-pushed the reduce-no-match-automation-telemetry branch from b27b878 to 37b292d Compare November 17, 2026 23:14
@malhotra5 malhotra5 changed the title Reduce no-match automation telemetry volume fix: reduce no-match automation telemetry volume Nov 17, 2026
@github-actions github-actions Bot added the type: fix A bug fix label Nov 17, 2026
Co-authored-by: openhands <openhands@all-hands.dev>
@malhotra5
malhotra5 merged commit 77652dc into main Nov 17, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants