fix(recall): fill the source_facts budget in rank order and flag truncation (#3221)

[nicoloboschi]

Fixes #3221.

With include.source_facts enabled, recall returns each observation's full source_fact_ids but resolves them through a source_facts map filled up to a token budget. All three problems reported in #3221 are real; this fixes them.

1. The budget was spent in DB row order, not rank order

observation_ids is built in rank order from top_scored, but the follow-up read discarded that order:

SELECT id, source_memory_ids FROM memory_units
WHERE id = ANY($1::uuid[]) AND fact_type = 'observation'   -- no ORDER BY

So whichever observation Postgres happened to return first spent the budget first, and a rank-1 result could lose its provenance to a rank-90 one — differently on two banks with identical content. The sibling enrichment paths in the same function already avoid this (chunks order by array_position(...), entities preserve top_scored order); source facts were the one that didn't. Both branches (SQL store and store-owned rows) now keep observation-rank order, so truncation hits the tail of the result list and never the head.

2. One oversized fact evicted everything behind it

The budget loop breaked on the first fact that didn't fit, so a single long fact dropped every shorter fact after it. It now skips just that fact and keeps filling.

3. Truncation was silent

results[].source_fact_ids still advertises every source, so a caller could not tell a budget-truncated map from a dangling reference. The response now carries source_facts_truncated: true when the budget dropped something (present only when source facts were requested). source_fact_ids is left complete on purpose — callers keep the full provenance list and the flag tells them why an ID may be unresolvable.

Implementation

The budget fill moved out of the 700-line recall body into engine/source_facts.py (select_source_facts_within_budget, returning a SourceFactSelection dataclass) so the selection rules are directly testable. Behaviour is otherwise unchanged: same dedup, same per-observation cap semantics, same defaults.

Tests

• tests/test_source_facts_selection.py — 9 unit tests over the selection rules: rank-order fill, oversized-fact skip (both budget modes), dedup of shared sources, unresolvable ID is not reported as truncation.

• tests/test_source_facts_tokens.py::TestRecallSourceFactsRankOrder — end-to-end regression: budgets the top result's sources exactly and asserts it keeps them. Verified it fails without the fix, with exactly the reported symptom:

  AssertionError: the top-ranked result lost provenance to a lower-ranked one
  

Existing source-fact, recall-enrichment and observation tests pass unchanged.

Also updated

OpenAPI + generated Go/Python/TypeScript clients, the recall API docs, and the docs skill.