Release 0.7.11: Claude Platform on AWS, usage and cache reporting, extended thinking - #166
Merged
Conversation
Per CLAUDE.md 'Develop Branch Sync Rule': after each alpha release on main (0.7.10a2 here), develop bumps the patch and resets to .dev0 so the two branches share a clean version lineage and dev builds publish to TestPyPI under the next patch series. Fast-forwarded merge from main (no divergence: develop had nothing ahead). All #146 (persistent hed-lsp) and #151 (#148+#150 latency) work is now on develop.
Server mode reads ANTHROPIC_API_KEY/ANTHROPIC_BASE_URL/ANTHROPIC_WORKSPACE_ID (workspace header required by the AWS endpoint); BYOK keys go to the first-party API. Offered models: claude-haiku-4-5 (default) and claude-sonnet-5. Replaces langchain-openai/litellm deps with langchain-anthropic. Verified live against both models.
- create_anthropic_workflow/create_byok_workflow/create_vision_agent replace the OpenRouter factories; provider routing and user-id cache lanes removed - BYOK accepts Anthropic keys (sk-ant-) via X-Anthropic-Key; legacy X-OpenRouter-Key header still accepted as transport - Unknown models rejected with 400; server mode no longer needs a per-request key (credentials come from the environment) - Ollama path removed; evaluation judge defaults to Claude Haiku 4.5 - Feedback triage and telemetry defaults updated
- Defaults: claude-haiku-4-5 for annotation, evaluation, and vision; claude-sonnet-5 offered via --model - Provider options removed (single provider now); credentials store anthropic_api_key with HEDIT_ANTHROPIC_API_KEY env override - Standalone mode can run on ANTHROPIC_API_KEY env credentials without a stored BYOK key; client sends X-Anthropic-Key
…thropic-Key Provider plumbing removed from the web UI (single provider now); model values use first-party ids. Worker BYOK detection and header forwarding accept the new X-Anthropic-Key header alongside the legacy one.
- New test_anthropic_llm.py unit tests (normalization, server/BYOK modes, temperature and thinking gating, caching wrapper) - test_integration_openrouter.py renamed to test_integration_anthropic.py, gated on ANTHROPIC_API_KEY - Security, CLI, and endpoint tests updated for X-Anthropic-Key and first-party model ids; removed tests of deleted OpenRouter modules - 440 non-integration tests pass (LSP tests fail locally for an unrelated hed-lsp version issue)
CI now uses the ANTHROPIC_API_KEY secret plus ANTHROPIC_BASE_URL and ANTHROPIC_WORKSPACE_ID repository variables (must be configured on GitHub).
Factory raises RuntimeError when ANTHROPIC_API_KEY is unset in server mode; endpoints map it to 503 while model validation errors stay 400.
- New docs/deployment/claude-platform-aws.md replaces openrouter.md - Ollama/GPU setup removed from deployment guides; Anthropic credential setup and model selection (claude-haiku-4-5 default, claude-sonnet-5 optional) documented throughout - BYOK manual tests rewritten for X-Anthropic-Key with sk-ant keys
- 400 mapping for rejected models on all four annotate endpoints; exact 503 with missing server credentials; alias acceptance over HTTP - BYOK key extraction via both X-Anthropic-Key and legacy header - Standalone-mode credential gate (env credentials, no credentials, API mode still requires key) - Replace stale pre-migration telemetry test data; pin the BYOK format-rejection test to 401
Critical: - BYOK LLMs now pin base_url to api.anthropic.com explicitly; ChatAnthropic otherwise inherits the server's ANTHROPIC_BASE_URL from the process env, routing BYOK keys to the AWS endpoint that rejects them (verified live) - Anthropic exceptions map to specific HTTP statuses via a shared classifier across all four annotate endpoints (401 auth, 403 permission, 413 context overflow, 400 bad request, 502 connection, 504/429 kept) Robustness: - Startup validates credentials with a free count_tokens call; failure keeps the server up but marks /health degraded instead of booting a healthy-looking server that 500s on every request - Vision-agent init failure no longer takes down text annotation (503 on image endpoints only) - CachingLLMWrapper raises TypeError on unsupported message types instead of silently relabeling them as user turns - CLI warns once when a legacy OpenRouter key is found in credentials; client gains explicit 400/429 error branches - Frontend streaming handlers surface the backend's error detail instead of a bare HTTP status Cleanup: dead LLM_PROVIDER_PREFERENCE read removed, user_id docstrings no longer claim telemetry recording, obsolete OpenRouter benchmark examples deleted, worker forwards X-OpenRouter-Vision-Model, stale docstrings and comments corrected. All 454 unit tests pass; lifespan integration tests pass against the live endpoint.
…m-aws Migrate LLM stack to Anthropic Claude via Claude Platform on AWS
CachingLLMWrapper now reports each response's usage metadata to a new usage ledger, labeled with the agent role that made the call. The ledger keeps process totals (for server metrics) and per-request totals via a contextvar scope that covers LangGraph's concurrent nodes. Cost is computed from Anthropic list prices with the cache multipliers (read 0.1x, 5-minute write 1.25x), alongside what the same calls would have cost uncached, which is what makes a savings figure meaningful. Unpriced models contribute token counts and are counted separately rather than silently costing zero. Tested: 50 unit tests covering the token split (generic and per-TTL cache keys), cost and savings math, role/model breakdown, scope isolation including concurrent tasks, and extraction from real AIMessage and ChatResult objects.
Annotation endpoints (streaming included) now run inside a usage scope, so every LLM call the workflow makes is attributed to the request that caused it. The figures land in three places: - a `usage` field on annotation responses and on the stream's result event, which is what lets a caller see the cache savings on their own request - telemetry events, whose input/output token and cost fields were declared but never populated, plus new cache-read/write, hit-rate, call-count, and uncached-cost fields - GET /metrics, reporting server-wide totals by role and by model since startup; BYOK callers get 403 since their own numbers come back on their annotation response Request-override headers also gained X-Anthropic-* spellings (model, eval-model, vision-model, temperature, key) behind one helper, with the legacy X-OpenRouter-* names still accepted as transport. Tested: header precedence and both spellings reaching model validation, /metrics auth and breakdowns, telemetry population from a ledger. 108 tests in the touched files pass.
The CLI now reports, under the annotation, how many tokens the request used, how many came from the prompt cache, what it cost, and how much the cache saved. API mode reads the figures from the response's usage field; standalone mode collects them locally in a usage scope covering the vision call and the whole workflow. Two defects in the same output path are fixed while wiring this up: - Standalone mode returned only `hed_string`/`description`, while the shared text renderer reads the API's field names, so `hedit annotate --standalone` printed an empty annotation and no status detail. The result is now shaped with both spellings. - The status line built Rich markup and appended it with Text.append, which prints markup literally: users saw "[green][x] Valid[/]". It is now rendered as markup, with the ASCII checkboxes escaped so they survive. Tested: 16 tests over line formatting (cache hit, first run, singular call, unpriced model, sub-cent costs), panel rendering, JSON passthrough, markup escaping, and the standalone result shape.
Measured against the AWS endpoint with count_tokens: the annotation system prompt is 21,811 tokens, while evaluation (623), assessment (266), feedback (241), keyword (186), and the vision prompt (51) sit far below Haiku 4.5's 4096-token minimum cacheable prefix. So annotation is the only role that can cache; the markers on the others are accepted but never create an entry, and cost nothing extra. Universal caching in the "every role caches" sense is not reachable by adding markers, and inflating prompts to reach the minimum would cost more than it saves. A real two-request run shows the economics: the first request writes the prefix at the 1.25x premium ($0.0299 vs $0.0245 uncached), the second reads it back for $0.0047, 81% below uncached. Break-even is two requests inside the 5-minute window, so interactive use spaced further apart pays the write premium repeatedly; cache_ttl (or HEDIT_PROMPT_CACHE_TTL) now accepts "1h" for that traffic shape, with the 5-minute default unchanged. Tested: live cache write-then-read and short-prefix-does-not-cache tests against the AWS endpoint (both pass), cached-prefix byte-stability tests including the no_extend fork, and the TTL marker/validation unit tests.
The CLI client, frontend, and worker now send the X-Anthropic-* spelling of the per-request overrides. The API accepts both spellings, and the worker forwards both, so nothing breaks for cached frontends or third-party clients still sending X-OpenRouter-*. Deployment order matters for this change: API first (it already accepts both), then the worker (it must forward the new names before anything sends them), then the frontend. The web UI also gained a "Usage and Cache Savings" section fed by the new usage field on the response, so the saving is visible where users actually run annotations. Tested: CLI client header tests updated to assert the new names and the absence of the old ones; worker JS syntax checked; renderUsage exercised with real response payloads (cache-hit and first-run shapes) and with null/zero-call input.
Adds docs/prompt-caching.md with the measured prompt sizes per role (only annotation's 21.8k-token prefix clears Haiku 4.5's 4096-token minimum), the two-request cost comparison showing the write premium and the 81% saving, the cache-lane rules that keep the prefix stable, the four places the figures are surfaced, and the live test that verifies the claim. Also documents HEDIT_PROMPT_CACHE_TTL in .env.example and the deployment guide, records the X-Anthropic-* header names as primary there, and points README and the docs index at the new page.
Sweeps the leftovers from the migration where the text or code still described the old stack: - SECURITY.md documented OpenRouter key handling and an Ollama local-GPU privacy story that no longer exists; it now covers the Anthropic credentials, BYOK keys, and the fact that every annotation leaves the machine. - .context/agent-architecture.md still credited Qwen-VL for vision and OpenRouter with an Ollama fallback; .context/api-and-deployment.md still documented provider-routing headers as live. - .rules/testing.md pointed at OPENROUTER_API_KEY_FOR_TESTING. - debug_workflow.py imported ChatOllama, was referenced by nothing, and could not run; deleted. - examples/test_examples.py had the same dead import plus a hardcoded schema path; ported to the Anthropic factory and now prints the cache savings per example. Verified live: 3/3 examples annotate successfully. - plan.md records the epic outcome and a current telemetry event example; the manual BYOK test doc uses the X-Anthropic-* header names. Remaining OpenRouter mentions are deliberate: legacy header acceptance, the credentials-file migration warning, tests asserting that back-compat, and historical benchmark data.
The six failing LSP tests were not a HEDit bug: the local hed-lsp checkout sits on a feature branch at 0.3.3, 32 commits behind origin/main, and predates the hed/suggest request handler (added in 0.4.0, commit c5a3f85). Its out/server.js was also built in February. The server answers "Unhandled method hed/suggest" with JSON-RPC -32601, which surfaced as six assertion failures instead of naming the real problem. Verified by building origin/main (0.4.0) in a scratch worktree and pointing HED_LSP_SERVER_JS at it: all 13 LSP tests pass, and the full suite is 526 passed / 0 failed. So the client is correct as written. The fixture's docstring already claimed it skipped when the hed/suggest endpoint was unavailable, but it never checked. It now detects a bundle without hed/suggest and skips with the rebuild command, so a stale dependency reads as a stale dependency. Also documents why extended thinking is disabled everywhere it can be: with thinking on, the agents emit far more text and tend to circle in reasoning loops rather than converging. Where a model refuses to disable thinking, the factory now requests the lowest reasoning effort instead; _ALWAYS_THINKING_MODELS is empty today, so the rule is in place for whichever model needs it first. Addresses the CodeQL unreachable-code alert on tests/test_llm_usage.py by raising inside a helper, so the control flow is visible to the scanner.
The previous wording claimed every role disables thinking. Annotation and vision do not: they take the model default, which is no thinking on Haiku 4.5 and adaptive thinking on Sonnet 5. Records the per-provider intent (on for Anthropic, off for others, where thinking was slow enough to erase the caching savings) and marks the annotation budget as an open measurement rather than a settled choice.
…al-caching-observability Complete Phase 2: prompt-cache measurement, usage reporting, and migration cleanup
hedtools reports issue severity as an ErrorSeverity enum (ERROR=1, WARNING=10), not the string "error". The Python validator compared it to "error", which is never true, so every error was filed as a warning and is_valid came back True for any input: "NotARealTag/Foo" validated clean, and the workflow's refinement loop could not fire because validation never failed. The JavaScript validator path was unaffected (the JS validator returns separate error and warning arrays). Anything not explicitly a warning now counts as an error, so an unrecognized severity fails closed rather than passing silently. Note this makes the refine loop live on the Python-validator path, so requests with genuinely invalid tags will now iterate where they previously returned on the first attempt. Tested: severity mapping for unknown tags, non-base tags, repeated expressions (all errors, is_valid False) and tag extensions (warning, is_valid True). test_validate_invalid_tag previously asserted "is_valid is False or warnings > 0" with a note claiming HED 8.3.0+ reports invalid tags as warnings; that note described this bug, and the test now asserts the error. 533 tests pass.
Adds a `thinking` parameter that overrides disable_reasoning, with
validation matching what the API actually enforces (verified against the
AWS endpoint):
- Haiku 4.5 has no adaptive mode and needs
{"type": "enabled", "budget_tokens": N}, N >= 1024 and below max_tokens.
- Sonnet 5 rejects thinking.type "enabled" ("Use thinking.type adaptive")
and takes {"type": "adaptive"} or {"type": "disabled"}.
- Enabling thinking drops `temperature`: the API returns 400
("temperature may only be set to 1 when thinking is enabled") otherwise.
- {"type": "disabled"} is accepted on both, so a caller can express "off"
uniformly instead of special-casing per model.
Needed to measure whether thinking earns its cost on the annotation agent;
the answer decides whether any default changes.
Tested: each accepted and rejected shape per model, the temperature drop
and its absence when thinking is disabled, budget bounds, and precedence
over disable_reasoning.
Measured six configurations over the 15 benchmark descriptions, varying only the annotation LLM (evaluation and keyword stayed on Haiku with thinking off): | arm | 1st-attempt valid | attempts | latency | cost/req | |----------------------------|-------------------|----------|---------|----------| | Haiku, no thinking | 5/15 | 1.87 | 10.3s | $0.0092 | | Haiku, no thinking, temp 1 | 6/15 | 1.73 | 10.0s | $0.0086 | | Haiku, 1024 budget | 11/15 | 1.27 | 22.8s | $0.0129 | | Haiku, 2048 budget | 13/15 | 1.13 | 20.9s | $0.0114 | | Sonnet 5, thinking off | 10/15 | 1.40 | 11.8s | $0.0261 | | Sonnet 5, adaptive | 12/15 | 1.20 | 12.9s | $0.0267 | Annotation now runs with a 2048-token budget on Haiku and adaptive on Sonnet 5, which has no budget mode. Notable results: 2048 costs less per request than 1024 because it removes more refinement rounds than it adds in thinking tokens; total LLM calls drop 71 -> 49; no reasoning loops appear (attempts fall); and Sonnet 5 reaches no higher first-attempt validity than Haiku-with-thinking at 2.3x the cost. A no-thinking arm at temperature 1.0 rules out the forced temperature change as the cause. The price is latency, roughly doubled, so HEDIT_ANNOTATION_THINKING_BUDGET=off turns it back off for latency-sensitive deployments. Support roles keep thinking off (#150). Non-Anthropic models, if HEDit gains them (#163), default to off: that is where thinking was slow enough to erase the caching savings. Adds examples/thinking_experiment.py so the measurement can be repeated when the default model changes (#64), the raw results, and docs/reasoning.md. Tested: policy resolution per model, env override and disable values, budget bounds, and one live end-to-end annotation confirming the workflow runs with thinking on and no reasoning text reaches the annotation string.
The thinking experiment showed Haiku 4.5 with a 2048-token budget matches Sonnet 5 on first-attempt validity at 2.3x less cost, so calling Sonnet "highest quality" in the UI, CLI, and docs is no longer accurate. Sonnet stays selectable everywhere it was, with the cost difference stated and the web app linking to the measurement. Tested: 555 unit tests pass; ruff clean.
docs/reasoning.md is not on main yet, so a blob/main link 404s until the next release. Switch to blob/develop when it lands on main.
Releases have been described only by the auto-generated commit list. This records 0.7.11 in prose: the Claude Platform on AWS migration, per-role token and prompt-cache accounting on every surface, the 1-hour cache TTL, extended thinking on annotation with its measured effect, the X-Anthropic-* headers, and the validator severity fix.
neuromechanist
temporarily deployed
to
testpypi
August 21, 2026 07:06 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Member
Author
Live verification before mergeThe full integration suite does not run on pull requests (its condition is a push or a manual dispatch), so it was dispatched on develop at this commit: run 32457792491. Including against the real endpoint:
The The 18 skips are the tests gated on credentials this runner does not carry (BYOK first-party keys). |
…exposure) The bad_request branch of _describe_llm_error forwarded str(exc)[:200] to the client, which CodeQL flagged on both streaming endpoints. The message is now a fixed string like every other branch. Nothing is lost operationally: all four call sites already log the exception, so the provider's own wording stays in the server log. Adds tests asserting no branch leaks exception text, and that context overflow still takes precedence over bad_request. Tested: 559 unit tests pass; ruff clean.
…text-in-responses Stop quoting provider errors in API responses (CodeQL)
Deploying hedit with
|
| Latest commit: |
162c87f
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://33ec7cf9.hedit.pages.dev |
| Branch Preview URL: | https://develop.hedit.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the develop to main release PR for the 0.7.11 line.
CHANGELOG.md(added in #165)carries the full notes; the summary below is what a reviewer should look at.
What ships
Provider migration (#155, #156, #158, #162)
All LLM calls go to the Claude Platform on AWS: the Anthropic-operated Messages API billed
through AWS Marketplace, not Bedrock. Offered models are
claude-haiku-4-5(annotation,evaluation judge, vision) and
claude-sonnet-5; Opus is not offered. OpenRouter, LiteLLM,and Ollama paths are gone; multi-provider support is tracked in #163. Legacy
OpenRouter-style model ids still resolve as aliases, and the
X-OpenRouter-*headers arestill accepted alongside the new
X-Anthropic-*spellings, so older clients and cachedfrontends keep working.
Usage, cost, and cache reporting (#155, #157, #164)
Per-request and per-role accounting in
src/utils/llm_usage.py, surfaced in the CLI, theusageobject on annotation responses, the web app, telemetry events, and a newGET /metricsendpoint (server key required; BYOK callers get 403 and read their ownnumbers from their response). Measured: a cache hit costs $0.004670 against $0.024295
uncached, an 81% saving, with break-even on the second request in the window.
HEDIT_PROMPT_CACHE_TTL=1hsuits interactive single-user traffic. Details indocs/prompt-caching.md.Extended thinking on annotation (#154, #165)
2048-token budget on Haiku 4.5, adaptive on Sonnet 5, off for the support roles and off by
default for any non-Anthropic model added later. Measured over 15 benchmark descriptions:
first-attempt validity 5/15 to 13/15, average attempts 1.87 to 1.13, LLM calls down a
third, cost up 24%, latency roughly doubled (10.3s to 20.9s).
docs/reasoning.mdhas thesix-arm table and
HEDIT_ANNOTATION_THINKING_BUDGET=offreverts it.Validator severity fix (#161)
hedtoolsreports severity as anErrorSeverityIntEnum and it was compared against thestring
"error", so every error was recorded as a warning,is_validwas always True, andthe refinement loop could never fire on the Python-validator path (CLI standalone runs and
servers without Node). The JavaScript validator path was unaffected. This is the fix that
made the thinking measurement meaningful: with the gate stuck open, every arm scores a
perfect first-attempt rate.
Deployment notes
ANTHROPIC_API_KEY,ANTHROPIC_BASE_URL, andANTHROPIC_WORKSPACE_ID; theendpoint rejects requests without the workspace header. First-party BYOK keys go to
api.anthropic.comwithout it.Both header spellings are accepted server-side, so the order is safe either way.
HEDIT_ANNOTATION_THINKING_BUDGET=offon a latency-sensitive deployment; final validityis unchanged either way, since the refinement loop converges.
0.7.11a1, tags it,and publishes to PyPI as a prerelease.
Verification
Standalone Tests (PR to main)lane ran a live annotation workflow against the realendpoint on this PR and passed, which exercises the severity fix and the thinking default
end to end.
a manual dispatch), so it was triggered manually on develop for this release; it last ran
green after Complete Phase 2: prompt-cache measurement, usage reporting, and migration cleanup #164 merged to develop (17 passed, 18 skipped), including the prompt-caching
write and read checks. It runs again automatically on the push to main.
attempt with no reasoning text in the output.
Still open after this
OPENROUTER_API_KEY_FOR_TESTINGsecret and repository variable(the variable exposes a key in plaintext) and revoke those OpenRouter keys.
default matters.