Emit correlated observe-mode policy decision events

[sozercan]

Summary

Add an opt-in, content-free structured event for each schema-v2 policy-routing observation so operators can persist and correlate observe-mode shadow decisions after Vekil restarts.

This is a follow-up to #275.

Current behavior

Observe mode executes the configured baseline tier and classifies asynchronously. The classifier result is aggregated into GET /stats.json under policy_routing, including sampling/admission, drop reasons, classifier outcomes, latency/usage, and shadow_tiers.

The current telemetry is insufficient for per-request evaluation:

• policy statistics are in-memory and reset on restart;
• the completed request log records policy_decision=observe_baseline, not the later asynchronous shadow tier;
• no durable event correlates a shadow decision with the request's operation_id; and
• aggregate shadow-tier percentages cannot be joined to an external task-success or quality label.

Goals

• Emit one bounded structured event for each eligible observe request/outcome when explicitly enabled.
• Correlate the asynchronous observation with the ordinary completed-request log through the existing proxy-owned operation_id.
• Preserve the existing privacy and public-identity guarantees.
• Use existing stdout/structured logging so deployments can persist events through journald, Loki, CloudWatch, Datadog, or another external collector.
• Keep /stats.json aggregation unchanged.
• Add no in-process observer queue, backlog, or persistence database.

Proposed operator interface

Add an explicit opt-in, for example:

--policy-observe-decision-log
POLICY_OBSERVE_DECISION_LOG=true

The exact spelling may change during implementation, but the feature must remain disabled by default because an observe_sample_rate of 1.0 can produce one event per eligible request.

Proposed event

Use the existing JSON logger with a stable message/event name such as policy observe decision.

Example shape:

{
  "msg": "policy observe decision",
  "operation_id": "op_...",
  "profile": "coding-economy",
  "traffic_bucket": "medium_tools",
  "observe_outcome": "completion",
  "baseline_tier": "lightweight",
  "shadow_tier": "powerful",
  "classifier_latency_ms": 420,
  "physical_classifier_sends": 1,
  "message_count": 6,
  "tool_count": 2,
  "input_bytes": 8400,
  "truncated": false,
  "config_generation": "...",
  "profile_generation": "...",
  "classifier_generation": "...",
  "binary_generation": "..."
}

observe_outcome must distinguish at least:

• not_sampled
• profile_capacity
• global_capacity
• breaker_open
• completion
• unavailable
• uncertain
• abstain
• timeout
• rate_limited
• upstream_rejected
• invalid_output
• canceled

A completed classifier result may additionally include the existing bounded classifier enums (turn_type, code_scope, risk_level, and abstain) if the privacy review accepts them. It must never include free-form rationale.

Correlation and lifecycle requirements

• Capture the existing operation_id before observe classification detaches from the request context.
• The event may arrive after the ordinary request-completed log; consumers join by operation_id.
• Emit at most one terminal observe event per eligible request.
• Cancellation and shutdown must not produce duplicate or partially populated events.
• Logging failure must not affect request routing, response delivery, breaker state, or policy statistics.
• The implementation must not add an unbounded goroutine, queue, or retained event backlog.

Privacy and identity requirements

The event must not contain:

• prompt or message text;
• raw classifier output or rationale;
• tool schemas, tool arguments, or tool results;
• provider, deployment, internal route, or internal target IDs;
• credentials or inbound authorization;
• provider state, replay IDs, response IDs, or session identifiers;
• raw User-Agent or upstream request IDs.

Only the public policy profile ID and bounded counts/enums/hashes may be logged. Existing generation hashes must continue to exclude secret values.

Non-goals

• Built-in database or local durable storage
• Running the shadow terminal tier in production
• Persisting prompts or classifier responses
• A dedicated dashboard UI
• Replacing the existing /stats.json policy aggregates
• Claiming that observe percentages alone prove routing quality

Acceptance criteria

• Opt-in flag/environment configuration is documented and disabled by default.
• Successful observe classifications emit a correlated event with baseline and shadow tiers.
• Not-sampled, admission-drop, breaker, failure, uncertainty, and abstention paths emit the correct bounded outcome.
• Event generation does not alter the baseline execution plan in observe mode.
• The event carries public identity, traffic bucket, bounded counts, latency/usage where available, and generation hashes.
• The event uses the same operation_id as the corresponding request-completed log.
• No forbidden content or internal identity is present in serialized events.
• Logging remains fail-open and does not block response delivery.
• Concurrency, cancellation, shutdown, and race-detector tests pass.
• /stats.json behavior and existing policy counters remain unchanged.
• Documentation states that persistence is owned by external log infrastructure and that /stats.json resets on restart.

Validation

Add focused tests for:

• sampled completion and mapped shadow tier;
• deterministic not-sampled outcome;
• profile/global admission drops;
• breaker-open and classifier failure categories;
• abstain/uncertain mapping;
• correlation with the request summary operation ID;
• asynchronous completion after the baseline response;
• cancellation and shutdown;
• disabled logging producing no event;
• redaction/privacy invariants; and
• go test -race coverage for observe workers and logging.