Epic: Anthropic-native (AWS-billable) default + Haiku-for-all + universal prompt caching

[neuromechanist]

Summary

Make Anthropic (native Messages API) the shipped default provider for every model
role in HEDit — annotation, evaluation, assessment, feedback, keyword extraction, and
vision — retiring the qwen/qwen3.5-122b-a10b (via alibaba) eval/vision defaults.
OpenRouter and Ollama remain fully available as opt-in providers; only the default
flips.

This is funded by direct Anthropic API access billed to an AWS account (the May-2026
"charge Anthropic API cost to your AWS bill" path — distinct from Bedrock). Crucially,
that AWS path is not a new integration: it is the native Anthropic provider pointed
at a different base URL plus a workspace id. So the work is "add a first-class
anthropic provider," and AWS billing falls out as two optional overrides
(base_url, workspace_id) on that provider.

Owner decision (I am the HEDit maintainer/owner): flip the default. This is a
breaking change for users who currently rely on the implicit OpenRouter default;
see Migration below.

Motivation

• Direct Anthropic access (AWS-billed) removes the OpenRouter middle-hop for the
  Anthropic models HEDit already prefers for annotation, and gives native prompt
  caching and native thinking control instead of OpenRouter's portable shims.
• Consolidating all roles on Claude removes the cross-vendor qwen/alibaba
  dependency for eval and vision (Claude models are multimodal, so vision no longer
  needs a separate vendor).
• Keeping OpenRouter/Ollama available preserves BYOK and local-GPU deployments.

Current state (grounded)

Single hardwired upstream. create_openrouter_llm() (src/utils/openrouter_llm.py:15)
bakes in six OpenRouter-specific things: the openrouter/ LiteLLM prefix, the
OPENROUTER_API_KEY env, the OpenRouter app headers, the provider={"only":[...]}
routing knob, the portable reasoning.enabled=false knob, and OpenRouter's model
namespace (anthropic/claude-haiku-4.5).

• Credentials: CredentialsConfig has only openrouter_api_key
  (src/cli/config.py:54-56); env override at config.py:161-163.
• Defaults: DEFAULT_MODEL="anthropic/claude-haiku-4.5",
  DEFAULT_EVAL_MODEL/DEFAULT_VISION_MODEL="qwen/qwen3.5-122b-a10b" with
  alibaba providers (config.py:38-49); mirrored in ModelsConfig
  (config.py:59-77).
• Standalone LLM build: LocalExecutionBackend calls create_openrouter_llm
  for annotation/eval/keyword (src/cli/local_executor.py:174-223) and vision
  (local_executor.py:278-290); models come from config via get_executor
  (src/cli/main.py:203-233).
• Server LLM build: api/main.py:166-215 builds the five role LLMs; the startup
  path already branches on LLM_PROVIDER (openrouter vs ollama,
  api/main.py:437-500) and reads env model overrides (ANNOTATION_MODEL,
  EVALUATION_MODEL, VISION_MODEL, *_PROVIDER). .env.example already exposes
  ALLOW_BYOK=true (line 25) and LLM_PROVIDER=openrouter (line 44).
• Reasoning: annotation keeps reasoning on; eval/assessment/feedback/keyword pass
  disable_reasoning=True → OpenRouter reasoning.enabled=false
  (openrouter_llm.py, see #150).
• Caching: CachingLLMWrapper (openrouter_llm.py:100) adds
  cache_control: ephemeral to system messages; auto-enabled by is_cacheable_model()
  which matches model.startswith("anthropic/claude-").

Proposed design

1. A first-class anthropic provider (third LLM_PROVIDER value)

Add anthropic alongside openrouter/ollama. A provider owns exactly the axes that
differ:

Axis	openrouter	anthropic (native)
LiteLLM prefix	openrouter/<model>	anthropic/<model>
API key env	OPENROUTER_API_KEY	ANTHROPIC_API_KEY
Base URL	implicit	ANTHROPIC_BASE_URL (unset = api.anthropic.com; AWS = https://aws-external-anthropic.<region>.api.aws)
Extra headers	HTTP-Referer, X-Title	workspace header from ANTHROPIC_WORKSPACE_ID (via extra_headers) when set
Model ids	anthropic/claude-haiku-4.5	claude-haiku-4-5 (bare, dots→dashes)
Sub-provider routing	provider={"only":[...]}	n/a (ignored)
Reasoning off	reasoning.enabled=false	omit thinking (off by default)
Reasoning on	provider default	thinking={"type":"enabled","budget_tokens":N}

AWS billing = the anthropic provider with base_url + workspace_id set. No
special-casing. LiteLLM's anthropic/ provider accepts api_base and extra_headers,
so the exact wire header name for the workspace id (confirm against the AWS-external
docs) is carried through extra_headers.

Implementation shape: a provider-dispatching factory, e.g. create_llm(provider, model, api_key, base_url=None, extra_headers=None, disable_reasoning=False, ...) in
src/utils/ that returns the LiteLLM model wired for the chosen provider, wrapping
CachingLLMWrapper when applicable. Keep create_openrouter_llm as the OpenRouter
branch (called by the new dispatcher) — no back-compat shim, just internal reuse.

2. New shipped defaults (all native Claude)

Role	New default (native id)	Was
Annotation	claude-haiku-4-5	anthropic/claude-haiku-4.5 (OpenRouter)
Evaluation / assessment / feedback	claude-haiku-4-5	qwen/qwen3.5-122b-a10b (alibaba)
Keyword extraction	claude-haiku-4-5	annotation model (OpenRouter)
Vision	claude-haiku-4-5 (multimodal)	qwen/qwen3.5-122b-a10b (alibaba)

DEFAULT_PROVIDER becomes anthropic; eval_provider/vision_provider become unused
on the Anthropic path (they only mean something for OpenRouter routing). Opus etc. stay
reachable via override (--model claude-opus-4-5 or config), which is what the
word-blurb 1000-image run needs.

3. Config / credentials

Generalize CredentialsConfig to per-provider creds and add provider selection +
Anthropic connection fields:

# credentials.yaml (0600)
openrouter:
  api_key: ...
anthropic:
  api_key: ...
  base_url: https://aws-external-anthropic.us-east-2.api.aws   # optional; AWS billing
  workspace_id: wrkspc_...                                      # optional; AWS billing

Env overrides (these match the standard Anthropic SDK names, so reading them directly is
idiomatic): ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL, ANTHROPIC_WORKSPACE_ID,
plus existing OPENROUTER_API_KEY. Add active-provider selection to config
(e.g. models.provider_backend: anthropic | openrouter | ollama, or reuse
LLM_PROVIDER semantics for parity with the server).

4. Server (api/main.py)

Add an LLM_PROVIDER=anthropic branch mirroring the openrouter branch
(api/main.py:437-500): require ANTHROPIC_API_KEY, read optional
ANTHROPIC_BASE_URL / ANTHROPIC_WORKSPACE_ID, default the five role models to
claude-haiku-4-5, build via the new dispatcher. Build the vision LLM from Claude too
(drop the OpenRouter-only guard at api/main.py:493). This is where the maintainer's
AWS-billed deployment sets ANTHROPIC_BASE_URL + ANTHROPIC_WORKSPACE_ID in server env
— do not hardcode the maintainer's workspace id into the shipped CLI.

5. CLI (cli/main.py)

• hedit init: add --provider {anthropic|openrouter|ollama} (default anthropic),
  --base-url, --workspace-id; write into the per-provider credentials block
  (main.py:337-343). Update help text (currently points only to openrouter.ai/keys,
  main.py:322).
• hedit config set already supports dot-keys (main.py:813); ensure the new fields
  are settable.

Correctness notes / gotchas

1. Prompt caching silently disables on the new default. is_cacheable_model()
   only matches anthropic/claude- (OpenRouter namespace). Native ids are bare
   claude-..., so caching auto-enable returns False — losing the ~90% cache
   savings on the large HED guide. Fix: also match bare claude- ids.
2. Model-id namespace. All defaults, CACHEABLE_MODELS, and OPENROUTER_MODELS
   maps use OpenRouter names. Native Anthropic wants claude-haiku-4-5,
   claude-opus-4-5, etc. Either keep per-provider model maps or store native ids in
   config when provider=anthropic.
3. Vision is multimodal on Claude — no separate vision vendor needed; the
   OpenRouter-only vision guard (api/main.py:493) must be lifted.
4. provider/eval_provider/vision_provider config fields are OpenRouter-only.
   On the anthropic path they must be ignored (not forwarded as a provider={"only"}
   routing knob).
5. Reasoning translation. On native Anthropic, extended thinking is off unless
   thinking is passed. So disable_reasoning=True → simply omit thinking; the
   "annotation keeps reasoning on" behavior (#150) must be re-expressed as an
   explicit thinking={"type":"enabled","budget_tokens":N} if we want to preserve it
   (see Decisions).
6. Haiku 4.5 min-cacheable prefix is 4096 tokens — the comprehensive guide clears
   it, so native caching is fine (and cleaner than through OpenRouter).

Files to touch

• src/cli/config.py — per-provider CredentialsConfig; add active-provider +
  base_url/workspace_id; flip DEFAULT_* constants to native Claude ids;
  env loading for ANTHROPIC_* (config.py:38-77, 150-190, 161-163).
• src/utils/ — new create_llm(provider, ...) dispatcher; native-Anthropic
  branch (prefix/key/base_url/headers/reasoning); fix is_cacheable_model();
  per-provider model-id handling (openrouter_llm.py).
• src/cli/local_executor.py — build the four LLMs via the dispatcher from the
  active provider (local_executor.py:174-223, 278-290).
• src/api/main.py — LLM_PROVIDER=anthropic branch; Claude vision; env-driven
  Anthropic models (api/main.py:166-215, 437-500).
• src/cli/main.py — init/config provider flags + help
  (main.py:322, 337-343, 203-233).
• src/agents/vision_agent.py — confirm it is model-agnostic (takes an llm); no
  change expected beyond the default it receives.
• .env.example — make Anthropic the primary documented block
  (ANTHROPIC_API_KEY, optional ANTHROPIC_BASE_URL/ANTHROPIC_WORKSPACE_ID,
  Claude model ids); move OpenRouter/qwen to an "alternative provider" block;
  note LLM_PROVIDER=anthropic (.env.example:44-90).
• Docs — README.md, DEPLOYMENT.md, CLAUDE.md: default-provider language,
  key acquisition (console.anthropic.com), AWS-billing note.
• Tests — provider dispatch (openrouter/anthropic), native caching enablement,
  is_cacheable_model for bare claude- ids, config/creds round-trip, env
  override precedence. Real calls per repo policy (no mocks); gate any live
  Anthropic call behind a key-present skip.

Migration (breaking change)

• The implicit OpenRouter default is replaced by Anthropic. Users must either set
  ANTHROPIC_API_KEY (new default) or explicitly select --provider openrouter
  (or LLM_PROVIDER=openrouter) and keep their OPENROUTER_API_KEY.
• Existing credentials.yaml with a flat openrouter_api_key should be read and
  migrated into the openrouter: block on load (one-time, in-place), so current users
  are not hard-broken — this is a config-format migration, not a runtime
  compatibility shim.
• qwen/alibaba are no longer defaults but remain reachable via OpenRouter.

Decisions for the owner

1. Shipped default base URL — recommend vanilla api.anthropic.com (no override)
   for BYOK users; AWS-external base_url + workspace_id set only in the maintainer's
   server env. Confirm.
2. Annotation extended thinking — keep it on (small budget_tokens) to preserve the
   #150 first-attempt-quality behavior, or ship all-roles-off for determinism/cost and
   revisit? Recommend: on for annotation with a modest budget, off for the rest.
3. Model tier — Haiku 4.5 as the shipped default for all roles (Opus via override),
   confirmed? word-blurb's batch run overrides annotation to Opus.
4. Active-provider knob naming — reuse LLM_PROVIDER end-to-end (server + CLI) for
   one concept, or a separate CLI config key?

Acceptance criteria

• anthropic is a first-class provider on both server (LLM_PROVIDER=anthropic)
  and CLI (--provider anthropic), default for all five roles + vision.
• With only ANTHROPIC_API_KEY set, a standalone annotation runs end-to-end on
  Claude (annotation + eval + vision), no OpenRouter/qwen involved.
• Setting ANTHROPIC_BASE_URL + ANTHROPIC_WORKSPACE_ID routes to the AWS-external
  endpoint (billing to AWS) with no other code change.
• Native prompt caching is active on the default (is_cacheable_model true for bare
  claude- ids; cache-write/read observable).
• --provider openrouter and LLM_PROVIDER=ollama still work unchanged.
• Existing flat openrouter_api_key configs load without error (auto-migrated).
• Tests cover provider dispatch, caching enablement, and config/env precedence.