fix: respect explicit Accept: text/html from AI bot user agents

[abhay-codes07]

Summary

Bot user agents currently get the markdown twin even when they explicitly ask for Accept: text/html. The AEO spec says the opposite should happen, so this fixes the adapters to honor an explicit Accept while keeping the existing bot behavior for everything else.

Closes #69.

Changes

• Add shouldServeMarkdown(accept, isBot) to @dualmark/core. It serves markdown when RFC 7231 negotiation already picks markdown, or when the request is from a known bot and markdown is at least as acceptable as HTML. It checks the second case by negotiating with markdown listed first, so a tie (wildcard or empty Accept) resolves to markdown for bots, while an explicit Accept: text/html keeps the bot on HTML.
• Use the helper in the cloudflare, deno, fastly, netlify, nextjs, sveltekit, vercel and nuxt adapters, replacing the bot.isBot || fmt === "markdown" check that ignored the Accept header for bots.
• Add a regression test in each adapter: a GPTBot/1.0 UA with Accept: text/html now receives HTML, not a noindex markdown body.
• Add unit tests for the helper in @dualmark/core.

Behavior is unchanged for Accept: */*, no Accept, and Accept: text/markdown, so the existing conformance scores and bot handling stay the same.

Type

• Bug fix (non-breaking)
• New feature (non-breaking)
• Breaking change
• Spec change (requires bump in AEO_SPEC_VERSION)
• Docs / examples / tooling only

Verification

• bun run build passes
• bun run test passes (541 tests across the packages)
• bun run typecheck passes
• If touching examples: ran dualmark verify against the example end-to-end (no examples touched)
• If touching /spec/: ran sync-spec and committed the mirror (no spec files touched)

Changeset

• Added a changeset (@dualmark/core minor, adapters patch)

---

cc @thepushkaraj @aagarwal1012 for a look when you have a minute(file change might look scary but indeed needed have a look). Happy to adjust the approach if you would rather keep the decision logic inside each adapter instead of a shared core helper.

[vercel]

@abhay-codes07 is attempting to deploy a commit to the Dodo Payments Team on Vercel.

A member of the Team first needs to authorize it.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[thepushkaraj]

@dodo-squirrels review

[dodo-squirrels]

Review: looks good, approving the approach

I reviewed the change end-to-end and it's well-implemented and correct.

Core logic — shouldServeMarkdown(accept, isBot) in packages/core/src/negotiation.ts is a clean encoding of AEO spec §5. It serves markdown when (1) RFC 7231 negotiation already picks markdown, or (2) the request is from a known bot and markdown is at least as acceptable as HTML (checked by negotiating with markdown listed first so ties resolve to markdown). An explicit Accept: text/html where HTML strictly outranks markdown keeps the bot on HTML.

Edge cases verified:

• Accept: */*, text/*, and no Accept → still markdown for bots (unchanged behavior).
• Accept: text/markdown → markdown for any UA (unchanged).
• Browser-like Accept: text/html,...,*/*;q=0.8 from a bot UA → now correctly stays on HTML.
• Unacceptable types (e.g. Accept: image/png) are still handled by the pre-existing fmt === null && accept 406 guard, which runs before shouldServeMarkdown in every adapter — so no misrouting there.

Consistency — all 8 edge/framework adapters (cloudflare, deno, fastly, netlify, nextjs, nuxt ×2, sveltekit, vercel) were updated identically and each passes the raw accept header. The Astro adapter is correctly excluded since it does no UA-based negotiation.

Versioning — changeset is appropriate: @dualmark/core minor (new public export), adapters patch (bug fix).

Validation — checked out the branch and ran the suites locally:

• Full test suite: all 24 test tasks pass (incl. the new per-adapter and core shouldServeMarkdown tests).
• Typecheck: all 25 tasks pass, 0 errors.

No changes required. LGTM.