fix(tokens): count literal special-token text safely

[userInner]

Branch strategy

• Active development targets main.

What this PR does

Before this PR:

The exact o200k token-budget estimator used gpt-tokenizer's default special-token policy. Persisted or user-authored text containing a literal such as <|im_start|> therefore threw before the provider request was created, and the persisted error text could trigger the same failure on the next send.

After this PR:

The estimator counts special-token spellings as ordinary BPE text. It does not mutate message content or treat those spellings as encoder control tokens. Regression coverage exercises all five literals reported in the issue, both alone and inside surrounding prose.

Refs #19465

Why we need it and why it was done in this way

This tokenizer boundary estimates budgets over arbitrary user and persisted content; it is not the provider encoder. gpt-tokenizer exposes disallowedSpecial, so an empty set expresses the required semantics directly at the narrow adapter boundary and keeps every downstream budget consumer safe.

The following tradeoffs were made:

• Literal special-token spellings count as multiple ordinary text tokens, which is intentionally more conservative than counting each spelling as one control token.
• The options object and empty set are allocated once with the lazily loaded tokenizer.

The following alternatives were considered:

• Sanitizing or removing the literals would change user content.
• Allowing all special tokens would count the literals as control tokens and understate ordinary-text usage.
• Catching the exception and falling back to the heuristic tokenizer would make estimates depend on message content.

Links to places where the discussion took place: #19465 (comment)

Breaking changes

None.

Special notes for your reviewer

Validation completed locally:

• Confirmed the regression test fails on the previous implementation for <|im_start|>, <|im_end|>, and <|endoftext|>.
• pnpm exec vitest run --project main src/main/ai/tokens/__tests__ (55 tests)
• Full main-process Vitest suite
• pnpm typecheck:node
• Biome, Oxlint, ESLint, and git diff --check
• pnpm build

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

• Branch: This PR targets main
• PR: The PR description is expressive enough and will help future contributors
• Code: Write code that humans can understand and Keep it simple
• Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
• Upgrade: Impact of this change on upgrade flows was considered and addressed if required
• Documentation: A user-guide update was considered and is present (link) or not required. Check this only when the PR introduces or changes a user-facing feature or behavior.
• Self-review: I have reviewed my own code (e.g., via /gh-pr-review, gh pr diff, or GitHub UI) before requesting review from others

Release note

Fixed chat sends failing when message history contains literal special-token text.

[kangfenmao]

LGTM