Skip to content

fix: answer Slack's URL verification challenge on slack_v0 webhooks - #769

Open
all-hands-bot wants to merge 1 commit into
mainfrom
fix/slack-url-verification
Open

all-hands-bot wants to merge 1 commit into
mainfrom
fix/slack-url-verification

Conversation

@all-hands-bot

@all-hands-bot all-hands-bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

Slack's Events API requires URL verification: the first delivery to a newly
registered webhook URL is a signed {"type": "url_verification", "challenge": ...} payload, and Slack expects the raw challenge string back as the response
body. The event endpoint treated that delivery as an event, so verification
always failed:

  • with a body-derived delivery ID (event_id_expr, e.g. event.ts), the
    challenge payload has no such field and the endpoint answered 400 "Invalid
    body delivery ID";
  • even without it, the endpoint returned the EventResponse JSON instead of
    echoing the challenge.

After: a slack_v0 webhook answers a verified challenge with
PlainTextResponse(challenge) — after signature verification (the challenge is
signed like any other delivery), and before delivery-ID extraction and
admission policies that only real events can satisfy. A missing or non-string
challenge is rejected with 400. Every other scheme and all event processing is
unchanged.

No schema change; nothing to configure beyond the existing slack_v0 scheme
(see the schema field description, which now documents the handshake).

Issue Number

None — gap found while testing Slack events against a custom automation
webhook directly (HTTP) rather than through Socket Mode.

Testing

  • New tests in tests/test_event_router.py:
    • a signed URL verification echoes the challenge verbatim (text/plain,
      200), and nothing is recorded in integration_events;
    • a verification without a challenge is a 400;
    • an invalid signature gets 401 — the challenge is echoed only to a
      delivery Slack signed;
    • a normal signed event on the same webhook still parses, dedupes on
      event.ts and creates a run.
  • tests/test_event_router.py and tests/test_teams_webhooks.py pass fully;
    tests/test_providers.py shows no new failures (its three local failures
    here are SQLite-environment artifacts of the test harness, reproduced on
    main without this change — raw-SQL UUID binding requires the Postgres
    testcontainer).
  • ruff format and ruff check clean.

Compatibility and risk

  • Only slack_v0 custom webhooks are affected; built-in sources always use
    hmac_sha256_hex and never see this path. For a slack_v0 webhook whose
    sender genuinely delivers {"type": "url_verification", "challenge": ...}
    signed with the webhook secret, echoing the challenge is exactly the
    behavior the scheme's owner defines.
  • The echo happens after HMAC verification with the configured secret, so an
    unverified request can never extract a challenge probe.
  • Challenges are answered before event_id_expr/admission_expr; those
    policies are documented as event policies and cannot be applied to the
    handshake, which carries no event.

Jev-Fast-Audit

⚡ Jev fast audit · estimates · 0.30s · commit fb483b8
Strongest signal: No primary concern selected.
Evidence: No primary concern to locate.
Coverage: complete supplied coverage; 10/10 hunks, 4/4 files.

All estimates and evidence
Estimate Likelihood / value Direct evidence
SQL injection 4.0% No direct hunk selected
Command injection 3.0% No direct hunk selected
Weakened authentication 9.0% No direct hunk selected
Weakened authorization 11.0% No direct hunk selected
Contract regression 13.0% No direct hunk selected
Data loss 3.0% No direct hunk selected
Sensitive data disclosure 7.0% No direct hunk selected
Unexpected data transfer 3.0% No direct hunk selected
Credential misuse 9.0% No direct hunk selected
Untrusted instruction authority 2.0% No direct hunk selected
Package source redirection 3.0% No direct hunk selected
Unverified remote execution 2.0% No direct hunk selected
Privileged environment access 2.0% No direct hunk selected
Security assessment bypass 7.0% No direct hunk selected
Prohibited workload 2.0% No direct hunk selected
Primary concern None selected; confidence 80.0% No primary concern to locate

Slack's first delivery to a newly registered webhook URL is a signed
url_verification challenge that must be answered with the raw challenge
string. The event endpoint treated it as an event: body-derived delivery
ID extraction failed on the challenge (400), and even without it the
EventResponse JSON was returned instead of the challenge, so URL
verification always failed.

A slack_v0 webhook now echoes the challenge verbatim, after signature
verification and before delivery ID and admission policies only real
events satisfy.

Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions github-actions Bot added the type: fix A bug fix label Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Coverage

Warning

Your comment is too long (maximum is 65536 characters), so the coverage report was not added. See the job log for how to reduce it.

@all-hands-bot

Copy link
Copy Markdown
Contributor Author

CI unit-tests on this branch fails, but not from this change: main currently ships two migrations numbered 038 (038_add_run_deferred_until.py from #756 and 038_add_webhook_protocol_options.py from #715), so every alembic-based test fails with "Revision 038 is present more than once" and test_migration_history.py reports the collision. All 23 CI failures are in that pre-existing set; the event-router, providers and Teams-webhook suites that cover this PR's change pass, and locally tests/test_event_router.py is 30/30.

This PR adds no migration, so it neither causes nor repairs that collision - happy to renumber in a separate PR if useful.

This comment was created by an AI agent (OpenHands) on behalf of the user.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants