Merge pull request #166 from Annotation-Garden/develop

Release 0.7.11: Claude Platform on AWS, usage and cache reporting, extended thinking

Author: neuromechanist

.context/agent-architecture.md

@@ -68,7 +68,7 @@ HEDit uses a LangGraph-based multi-agent workflow to convert natural language ev
 
 ### 6. VisionAgent (`src/agents/vision_agent.py`)
 - **Input**: Base64-encoded image
-- **Process**: Uses vision-language models (e.g., Qwen-VL via OpenRouter)
+- **Process**: Uses a Claude model (natively multimodal; no separate vision vendor)
 - **Output**: Natural language description of image content
 - **Key feature**: Enables image-to-HED annotation pipeline
 
@@ -100,7 +100,12 @@ Defined in `src/agents/state.py` as `HedAnnotationState(TypedDict)`:
 
 ## Configuration
 
-- **LLM Provider**: OpenRouter API (production), with Ollama fallback
+- **LLM Provider**: Anthropic Claude via the Claude Platform on AWS (Anthropic-operated
+  Messages API, AWS Marketplace billing; not Amazon Bedrock)
+- **Offered models**: `claude-haiku-4-5` (default) and `claude-sonnet-5`
 - **Default model**: Configurable via environment/headers
-- **BYOK support**: Users can provide their own API keys
+- **BYOK support**: Users can provide their own Anthropic key (`X-Anthropic-Key`),
+  which routes to api.anthropic.com
+- **Prompt caching**: The annotation system prompt is cached; usage and savings are
+  reported per request (see `docs/prompt-caching.md`)
 - **Streaming**: LangGraph `astream_events` for real-time progress via SSE

.context/api-and-deployment.md

@@ -11,17 +11,23 @@
 - `POST /feedback`: User feedback submission
 - `GET /health`: Service health check
 - `GET /version`: Version information
+- `GET /metrics`: Token use, cost, and prompt-cache savings since startup
+  (server API key required; BYOK callers get 403)
 
 ### Authentication Modes
-1. **Server mode**: `X-API-Key` header (server's OpenRouter key)
-2. **BYOK mode**: `X-OpenRouter-Key` header (user's own API key)
+1. **Server mode**: `X-API-Key` header (server's Anthropic credentials are used)
+2. **BYOK mode**: `X-Anthropic-Key` header (user's own Anthropic key, `sk-ant-...`,
+   routed to api.anthropic.com)
 3. **Public endpoints**: `/feedback`, `/health`, `/version`
 
 ### Model Override Headers
-- `X-OpenRouter-Model`: Override annotation model
-- `X-OpenRouter-Vision-Model`: Override vision model
-- `X-OpenRouter-Provider`: Override provider routing
-- `X-OpenRouter-Temperature`: Override temperature
+- `X-Anthropic-Model`: Override annotation model
+- `X-Anthropic-Eval-Model`: Override evaluation model
+- `X-Anthropic-Vision-Model`: Override vision model
+- `X-Anthropic-Temperature`: Override temperature
+
+The legacy `X-OpenRouter-*` spellings of these names (plus `X-OpenRouter-Key`) are
+still accepted as transport; provider-routing headers are ignored.
 
 ### CORS
 - Production: `hedit.pages.dev`, `annotation.garden`

.env.example

@@ -19,8 +19,9 @@ API_KEYS=your-generated-api-key-here
 REQUIRE_API_AUTH=true
 
 # BYOK (Bring Your Own Key) Mode
-# When enabled, users can provide their own OpenRouter API key via X-OpenRouter-Key header
-# Their key is used for LLM calls (billing goes to their account)
+# When enabled, users can provide their own Anthropic API key (sk-ant-...)
+# via the X-Anthropic-Key header. Their key is used for LLM calls against
+# the first-party Anthropic API (billing goes to their account).
 # Set to "false" to require server API keys only
 ALLOW_BYOK=true
 
@@ -40,58 +41,63 @@ ALLOW_LOCALHOST_CORS=true
 # LLM Provider Configuration
 # ============================================================================
 
-# Provider: "openrouter" (recommended) or "ollama" (local GPU)
-LLM_PROVIDER=openrouter
-
 # Temperature: 0.0-1.0, lower = more consistent
 LLM_TEMPERATURE=0.1
 
 # ============================================================================
-# OpenRouter Configuration (when LLM_PROVIDER=openrouter)
+# Anthropic Configuration (Claude Platform on AWS)
 # ============================================================================
-
-# Get your API key from https://openrouter.ai/
-OPENROUTER_API_KEY=your-openrouter-api-key-here
-
-# Test API key (separate key to track testing costs)
-# Used only by integration tests that make real LLM calls
-# OPENROUTER_API_KEY_FOR_TESTING=your-testing-api-key-here
+# LLM calls go to Claude via the Claude Platform on AWS (Anthropic-operated
+# Messages API, AWS Marketplace billing; NOT Amazon Bedrock).
+# All three values are required in server mode; the endpoint rejects
+# requests without the anthropic-workspace-id header.
+# Key: AWS Console -> Claude Platform on AWS -> API keys -> Generate a key
+# Workspace: AWS Console -> Claude Platform on AWS -> Workspaces (wrkspc_...)
+ANTHROPIC_API_KEY=your-claude-platform-key-here
+ANTHROPIC_BASE_URL=https://aws-external-anthropic.us-east-2.api.aws
+ANTHROPIC_WORKSPACE_ID=wrkspc_your_workspace_id
 
 # ============================================================================
 # Model Configuration
 # ============================================================================
-# Each agent type has its own model and provider for optimal performance.
+# Offered models: claude-haiku-4-5 (default) and claude-sonnet-5.
 
-# Annotation Model (Claude Haiku 4.5 via Anthropic - high quality with prompt caching)
-ANNOTATION_MODEL=anthropic/claude-haiku-4.5
-ANNOTATION_PROVIDER=anthropic
+# Annotation Model (Claude Haiku 4.5 - fast, near-frontier quality)
+ANNOTATION_MODEL=claude-haiku-4-5
 
-# Evaluation/Assessment Model (Qwen3.5-122B via Alibaba - fast and cost-effective)
-# Used for evaluation, assessment, and feedback agents
-EVALUATION_MODEL=qwen/qwen3.5-122b-a10b
-EVALUATION_PROVIDER=alibaba
+# Evaluation/Assessment Model (the judge stays on Haiku)
+EVALUATION_MODEL=claude-haiku-4-5
 
-# Vision Model (Qwen3.5-122B via Alibaba - accepts vision, fast)
-VISION_MODEL=qwen/qwen3.5-122b-a10b
-VISION_PROVIDER=alibaba
+# Vision Model (Claude models are natively multimodal)
+VISION_MODEL=claude-haiku-4-5
+
+# Sonnet 5 is available for comparison. It matched Haiku 4.5 (which runs with
+# extended thinking) on first-attempt validity at 2.3x the cost, so it is not
+# recommended as a default; see docs/reasoning.md.
+# ANNOTATION_MODEL=claude-sonnet-5
 
 # ============================================================================
-# Alternative Models
+# Prompt Cache Lifetime
 # ============================================================================
-# For highest quality (but slower and more expensive):
-# ANNOTATION_MODEL=openai/gpt-5.2
-# ANNOTATION_PROVIDER=
-
-# For fastest results:
-# ANNOTATION_MODEL=anthropic/claude-haiku-4.5
-# ANNOTATION_PROVIDER=
+# The annotation system prompt (~21.8k tokens of HED vocabulary guide) is
+# cached, which cuts input cost by ~90% on a cache hit. Entries live 5
+# minutes by default, which suits server traffic; a cache write costs 1.25x,
+# so break-even is the second request inside the window.
+# Set to 1h for interactive use where requests are minutes apart (a 1-hour
+# write costs 2x and pays off from the third request in the hour).
+# See docs/prompt-caching.md
+# HEDIT_PROMPT_CACHE_TTL=5m
 
 # ============================================================================
-# Ollama Configuration (when LLM_PROVIDER=ollama)
+# Annotation Reasoning Budget
 # ============================================================================
-# Only used if LLM_PROVIDER=ollama (local GPU deployment)
-# LLM_BASE_URL=http://localhost:11435
-# LLM_MODEL=qwen2.5:32b
+# The annotation agent runs with extended thinking, which took first-attempt
+# validity from 5/15 to 13/15 on the benchmark descriptions, cut total LLM
+# calls by a third, and cost 24% more per request with roughly twice the
+# latency. Raise the budget, or set 0/off to disable it for a
+# latency-sensitive deployment. Ignored on Sonnet 5, which chooses its own
+# depth (adaptive). See docs/reasoning.md
+# HEDIT_ANNOTATION_THINKING_BUDGET=2048
 
 # ============================================================================
 # HED Configuration
@@ -169,6 +175,4 @@ LOG_LEVEL=INFO
 # GITHUB_REPOSITORY_OWNER=Annotation-Garden
 # GITHUB_REPOSITORY=hedit
 
-# Feedback processing uses OPENROUTER_API_KEY_FOR_TESTING (if set) to track
-# feedback-related LLM costs separately from annotation costs.
-# Falls back to OPENROUTER_API_KEY if testing key is not set.
+# Feedback processing uses the ANTHROPIC_API_KEY server credentials.

.github/workflows/docker-build.yml

@@ -33,8 +33,8 @@ jobs:
           docker run -d --name hed-bot-test \
             -p 38427:38427 \
             -e REQUIRE_API_AUTH=false \
-            -e LLM_PROVIDER=openrouter \
-            -e OPENROUTER_API_KEY=test-key-for-ci \
+            -e LLM_PROVIDER=anthropic \
+            -e ANTHROPIC_API_KEY=test-key-for-ci \
             -e LLM_TEMPERATURE=0.1 \
             hed-bot-test:ci
 

.github/workflows/process-feedback.yml

@@ -68,7 +68,9 @@ jobs:
         if: steps.feedback.outputs.has_files == 'true'
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY_FOR_TESTING }}
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          ANTHROPIC_BASE_URL: ${{ vars.ANTHROPIC_BASE_URL }}
+          ANTHROPIC_WORKSPACE_ID: ${{ vars.ANTHROPIC_WORKSPACE_ID }}
           DRY_RUN: ${{ github.event.inputs.dry_run || 'false' }}
         run: |
           # Process each feedback file

.github/workflows/release.yml

@@ -56,6 +56,8 @@ jobs:
           body: |
             ## HEDit ${{ steps.get_version.outputs.VERSION }}
 
+            Release notes in prose: [CHANGELOG.md](https://github.com/${{ github.repository }}/blob/main/CHANGELOG.md)
+
             ${{ steps.changelog.outputs.CHANGELOG }}
 
             ---
@@ -82,7 +84,7 @@ jobs:
 
             ### Quick Start
             ```bash
-            # Initialize with your OpenRouter API key
+            # Initialize the CLI (optionally with a BYOK Anthropic key)
             hedit init --api-key YOUR_KEY
 
             # Generate HED annotation

.github/workflows/test.yml

@@ -117,10 +117,10 @@ jobs:
           # - Integration test files (test_integration*.py, test_standalone*.py)
           # - Agent code (workflow, annotation, evaluation, keyword extraction, etc.)
           # - Validation code
-          # - OpenRouter/LiteLLM utility
+          # - Anthropic LLM utility
           # - CLI code (for CLI integration tests)
           # - Semantic search code
-          INTEGRATION_PATTERNS="tests/test_.*integration|tests/test_standalone|src/agents/|src/validation/|src/utils/openrouter|src/utils/litellm|src/utils/semantic|src/cli/"
+          INTEGRATION_PATTERNS="tests/test_.*integration|tests/test_standalone|src/agents/|src/validation/|src/utils/anthropic|src/utils/semantic|src/cli/"
 
           if echo "$CHANGED" | grep -qE "$INTEGRATION_PATTERNS"; then
             echo "integration_needed=true" >> $GITHUB_OUTPUT
@@ -160,13 +160,15 @@ jobs:
 
       - name: Run standalone tests only
         env:
-          OPENROUTER_API_KEY_FOR_TESTING: ${{ secrets.OPENROUTER_API_KEY_FOR_TESTING }}
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          ANTHROPIC_BASE_URL: ${{ vars.ANTHROPIC_BASE_URL }}
+          ANTHROPIC_WORKSPACE_ID: ${{ vars.ANTHROPIC_WORKSPACE_ID }}
         run: |
-          if [ -n "$OPENROUTER_API_KEY_FOR_TESTING" ]; then
+          if [ -n "$ANTHROPIC_API_KEY" ]; then
             echo "Running standalone tests..."
             pytest tests/ -v -m standalone --timeout=180 --cov=src --cov-report=xml:coverage-standalone.xml --cov-report=term-missing
           else
-            echo "OPENROUTER_API_KEY_FOR_TESTING not set, skipping standalone tests"
+            echo "ANTHROPIC_API_KEY not set, skipping standalone tests"
           fi
 
       - name: Upload standalone coverage to Codecov
@@ -185,7 +187,7 @@ jobs:
     needs: [check-changes]
     # Only run integration tests when:
     # 1. Push event (after merge) or manual trigger, not on PRs
-    # 2. Integration-related files changed (agents, validation, openrouter, or test file itself)
+    # 2. Integration-related files changed (agents, validation, anthropic, or test file itself)
     if: |
       (github.event_name == 'push' || github.event_name == 'workflow_dispatch') &&
       (needs.check-changes.outputs.integration_needed == 'true' || github.event_name == 'workflow_dispatch')
@@ -208,14 +210,16 @@ jobs:
 
       - name: Run integration tests only
         env:
-          OPENROUTER_API_KEY_FOR_TESTING: ${{ secrets.OPENROUTER_API_KEY_FOR_TESTING }}
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          ANTHROPIC_BASE_URL: ${{ vars.ANTHROPIC_BASE_URL }}
+          ANTHROPIC_WORKSPACE_ID: ${{ vars.ANTHROPIC_WORKSPACE_ID }}
         run: |
           # Only run if the secret is available
-          if [ -n "$OPENROUTER_API_KEY_FOR_TESTING" ]; then
-            echo "Running integration tests with OpenRouter..."
+          if [ -n "$ANTHROPIC_API_KEY" ]; then
+            echo "Running integration tests with Claude (Claude Platform on AWS)..."
             pytest tests/ -v -m integration --timeout=180 --cov=src --cov-report=xml:coverage-integration.xml --cov-report=term-missing
           else
-            echo "OPENROUTER_API_KEY_FOR_TESTING not set, skipping integration tests"
+            echo "ANTHROPIC_API_KEY not set, skipping integration tests"
           fi
 
       - name: Upload integration coverage to Codecov

.rules/testing.md

@@ -14,7 +14,8 @@ Never use mocks, stubs, or fake datasets. If real testing is not possible, don't
 
 ## When to Write Tests
 - **DO:** Test with real data and actual dependencies
-- **DO:** Use test API keys (`OPENROUTER_API_KEY_FOR_TESTING`)
+- **DO:** Use the Anthropic test credentials (`ANTHROPIC_API_KEY`, with
+  `ANTHROPIC_BASE_URL` and `ANTHROPIC_WORKSPACE_ID` for the Claude Platform on AWS)
 - **DO:** Test against actual file systems and real HED schemas
 - **DON'T:** Write tests if only mocks would work
 - **DON'T:** Create artificial test scenarios

.serena/memories/codebase_structure.md

@@ -22,7 +22,8 @@ hedit/
 │   │   ├── json_schema_loader.py  # JSON schema + vocabulary extraction
 │   │   ├── schema_loader.py       # Legacy Python schema loader
 │   │   ├── error_remediation.py   # Error augmentation for LLM feedback
-│   │   ├── openrouter_llm.py      # OpenRouter API integration
+│   │   ├── anthropic_llm.py       # Anthropic Claude integration (prompt caching)
+│   │   ├── llm_usage.py           # Token, cost, and cache accounting
 │   │   ├── litellm_llm.py         # Alternative LLM providers
 │   │   ├── image_processing.py    # Base64 image encoding
 │   │   └── github_client.py       # GitHub API for feedback issues

.serena/memories/project_overview.md

@@ -7,7 +7,7 @@ HEDit is a multi-agent system for converting natural language event descriptions
 - **Language**: Python 3.12+
 - **Package Manager**: uv
 - **Agent Framework**: LangGraph
-- **LLM Provider**: OpenRouter API (production), Ollama (fallback)
+- **LLM Provider**: Anthropic Claude via the Claude Platform on AWS (claude-haiku-4-5 default, claude-sonnet-5 optional)
 - **Validation**: HED JavaScript validator + HED Python tools (hedtools)
 - **Backend**: FastAPI
 - **CLI**: Typer + Rich