Fix validator severity mapping (#161) and add explicit reasoning control (#154)

[neuromechanist]

Summary

Fixes #161 (critical) and adds the reasoning knob from #154 that measuring the thinking
question required.

The validator bug

src/validation/hed_validator.py compared hedtools' severity against the string
"error", but hedtools reports an ErrorSeverity enum (ERROR=1, WARNING=10). That
comparison is false for every issue ever produced, so every error was filed as a warning,
errors stayed empty, and is_valid = len(errors) == 0 was always True:

'Sensory-event, NotARealTag/Foo'   -> is_valid=True  errors=0 warnings=1
'Agent-action, (Press, Button)'    -> is_valid=True  errors=0 warnings=1

Beyond the wrong is_valid, the refinement loop could never fire on this path:
validation never failed, so the workflow always returned after the first annotation
attempt. The multi-iteration architecture was inert wherever the Python validator runs -
CLI standalone mode (use_js_validator=False) and any server without Node or
HED_VALIDATOR_PATH. The JavaScript path was unaffected (it reads separate error and
warning arrays).

Now:

'Sensory-event, NotARealTag/Foo'      valid=False errors=1 warnings=0
'Agent-action, (Press, Button)'       valid=False errors=1 warnings=0
'Sensory-event, Visual-presentation'  valid=True  errors=0 warnings=0
'Sensory-event, Animal/Dog'           valid=True  errors=0 warnings=1  (TAG_EXTENDED)
'Sensory-event, Sensory-event'        valid=False errors=1 warnings=0  (repeated)

An unrecognized severity now counts as an error, so the validator fails closed.

Expect a behavior change: requests whose annotations really are invalid now iterate
instead of returning immediately, so those requests make more LLM calls. That is the loop
doing the job it was built for - visible immediately in a benchmark run, where cases that
previously reported "valid" on attempt 1 now take 2-3 attempts. max_validation_attempts
still bounds it.

test_validate_invalid_tag had encoded the bug as expected behavior, asserting
is_valid is False or len(warnings) > 0 under a note claiming "HED 8.3.0+ reports invalid
tags as warnings, not errors". That note described this bug; the test now asserts the
error.

The reasoning knob

A thinking parameter on create_anthropic_llm, overriding disable_reasoning, with
validation matching what the API enforces (each shape verified against the AWS endpoint):

Model	Accepted	Rejected
Haiku 4.5	{"type": "enabled", "budget_tokens": N} (N >= 1024, < max_tokens), {"type": "disabled"}	{"type": "adaptive"} - no adaptive mode
Sonnet 5	{"type": "adaptive"}, {"type": "disabled"}	"enabled" - "Use thinking.type adaptive"

Enabling thinking also drops temperature, since the API returns 400 ("temperature may
only be set to 1 when thinking is enabled").

No default changes here. Whether annotation should think is being measured across six
arms (Haiku with no thinking at temp 0.1 and temp 1.0, 1024- and 2048-token budgets,
Sonnet 5 with thinking off and adaptive) over the project's own 15 benchmark
descriptions, recovered from examples/model_benchmark.py. Results will be posted before
any default moves.

Testing

533 tests pass, including new coverage for the severity split (unknown tag, non-base tag,
repeated expression -> errors; tag extension -> warning) and for every accepted and
rejected thinking shape, the temperature drop, and budget bounds.

[cloudflare-workers-and-pages]

Deploying hedit with    Cloudflare Pages

Latest commit:	6b8a950
Status:	✅  Deploy successful!
Preview URL:	https://97603363.hedit.pages.dev
Branch Preview URL:	https://fix-validator-severity-161.hedit.pages.dev

View logs

[codecov]

Codecov Report

❌ Patch coverage is 97.56098% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/cli/local_executor.py	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[neuromechanist]

Experiment results: annotation now thinks

Six arms, same 15 benchmark descriptions, only the annotation LLM varying (evaluation and
keyword stayed on Haiku 4.5 with thinking off), max_validation_attempts=3:

Arm	Valid	Valid 1st attempt	Faithful	Attempts	Latency	Cost/req	Calls
Haiku, no thinking (temp 0.1)	15/15	5/15	5/15	1.87	10.3 s	$0.0092	71
Haiku, no thinking (temp 1.0)	14/15	6/15	5/15	1.73	10.0 s	$0.0086	66
Haiku, 1024 budget	15/15	11/15	6/15	1.27	22.8 s	$0.0129	53
Haiku, 2048 budget	15/15	13/15	7/15	1.13	20.9 s	$0.0114	49
Sonnet 5, thinking off	14/15	10/15	4/15	1.40	11.8 s	$0.0261	56
Sonnet 5, adaptive	15/15	12/15	6/15	1.20	12.9 s	$0.0267	51

Findings:

• Thinking more than doubles first-attempt validity, 5/15 to 13/15.
• 2048 tokens costs less per request than 1024 ($0.0114 vs $0.0129): the bigger budget
  removes more refinement rounds than it adds in thinking tokens. Input tokens across the
  run fell 635k to 387k, output rose 10.5k to 23.6k.
• Total LLM calls fall by a third (71 to 49); cost rises 24% over the no-thinking baseline.
• Latency roughly doubles. That is the real price, and
  HEDIT_ANNOTATION_THINKING_BUDGET=off turns it back off.
• No reasoning loops. Average attempts fall with thinking. The looping seen previously
  does not reproduce on Anthropic models with a working validator gate; it was an
  open-model observation, where thinking was also slow enough to erase the caching savings.
• Sonnet 5 is not worth its price here: adaptive reaches 12/15, no better than Haiku with
  thinking, at 2.3x the cost per request.
• Temperature is not the explanation: thinking forces temperature off, so a no-thinking
  arm at temperature 1.0 was run as a control (6/15 vs 5/15).
• Faithfulness barely moves (5/15 to 7/15).

Caveat: one run per arm at n=15, so a case or two of noise, and the 1024/2048 ordering
could flip. The 5-to-13 headline is well outside that.

These numbers only exist because the severity fix in this PR came first: with the
validator downgrading errors to warnings, is_valid was always True and every arm would
have scored a perfect first-attempt rate.

Added in this PR: the policy (annotation_thinking()), the harness
(examples/thinking_experiment.py) so it can be repeated when the default model changes
(#64), the raw results, and docs/reasoning.md.

[neuromechanist]

Model presentation follows the measurement

Default stays Claude Haiku 4.5 with a 2048-token thinking budget. Sonnet 5 remains selectable in the web app, the API (X-Anthropic-Model), and the CLI (--model) so the comparison can be reproduced rather than taken on trust, but it is no longer advertised as the quality option:

Surface	Before	After
Web app dropdown	Claude Sonnet 5 (highest quality)	Claude Sonnet 5 (larger, 2.3x the cost)
Web app	(no note)	help text: Haiku matched Sonnet on the 15-case benchmark for less than half the cost, linking to docs/reasoning.md
ALLOWED_MODELS, CLI help, .env.example, deployment docs	"highest quality"	cost stated, pointer to the measurement

docs/reasoning.md gains a Model choice section recording why the losing option stays visible.

Model ids and request handling are unchanged, so nothing about BYOK or the model allowlist moves. 555 unit tests pass, ruff clean.