Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,27 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.1.16] - 2026-02-23

### Added

- System prompt caching in the consensus pipeline for improved performance.
- MCP Client lifecycle monitoring with liveness guards and crash-to-error logging.

### Fixed

- Cost display budget timeouts and model attribution loss after child dismissal.
- Infinite continuation loop in fast-path consensus.
- `Decimal.decimal(nil)` crash in `Tracker.over_budget?` for N/A budgets.
- `Response.text()` blindness to pure JSON responses.
- Google Vertex string error body handling in RetryHelper v3.3.
- Silent result loss from HTTP actions missing timeout overrides.

### Changed

- DRY Aggregator SQL with extracted ResponseLogger module.
- Test suite optimization: removed ~230 redundant tests, split heavy modules, migrated 23 files from DataCase to ExUnit.Case.

## [0.1.15] - 2026-02-21

### Fixed
Expand Down
2 changes: 1 addition & 1 deletion lib/quoracle/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
- security/: Secret resolution and output scrubbing (added 2025-10-24)
- audit/: Secret usage tracking (added 2025-10-24)
- mcp/: Model Context Protocol client and configuration (added 2025-11-26)
- costs/: Agent cost tracking - AgentCost schema, Recorder, Aggregator (added 2025-12-13)
- costs/: Agent cost tracking - AgentCost schema, Recorder, Aggregator v3.0 with DRY SQL + per-model detailed queries (added 2025-12-13)

## Supervision Tree
```
Expand Down
12 changes: 11 additions & 1 deletion lib/quoracle/actions/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
- Spawn: Child agent spawning with downstream_constraints (475 lines, ConfigBuilder 273 lines, Helpers 82 lines, BudgetValidation 113 lines), dismissing flag check (v11.0), child_spawned notification (v12.0), profile parameter (v14.0), budget enforcement for budgeted parents (v17.0: :budget_required error), v19.0 removes Core.update_budget_committed callback (deadlock fix)
- Wait: Unified wait parameter (142 lines, true/false/number support, interruptible)
- SendMessage: Parent/child messaging (3-arity)
- DismissChild: Recursive child termination (249 lines, v4.0), async background dispatch to TreeTerminator, child_dismissed notification, budget reconciliation with absorption records and escrow release
- DismissChild: Recursive child termination (374 lines, v5.0), async background dispatch to TreeTerminator, child_dismissed notification, budget reconciliation with per-model absorption records and escrow-first release ordering
- Answer (263 lines): Gemini grounding search
- Model from ConfigModelSettings.get_answer_engine_model!() (config-driven)
- Raises RuntimeError if answer engine model not configured
Expand Down Expand Up @@ -133,6 +133,16 @@ Actions → Schema for parameter definitions
- **Router**: Added file_read, file_write routing via ActionMapper
- **CapabilityGroups**: file_read, file_write groups (allowed for all except restricted)

**Feb 23, 2026 - Cost Display & Budget Timeout Fix (WorkGroupID: fix-20260223-cost-display-budget-timeout)**:
- **DismissChild v5.0 (374 lines)**: Per-model absorption records + escrow-first ordering
- `create_absorption_records/7`: Creates one `child_budget_absorbed` record per model_spec (preserves model attribution)
- `reconcile_child_budget/8`: Escrow release BEFORE absorption records (prevents budget leak on parent crash)
- `build_absorption_metadata/4`: Includes model_spec, token counts, costs for LLM models; omits for external costs
- `query_child_tree_by_model/1`: Snapshots per-model tree costs before TreeTerminator deletes records
- `decimal_to_string/1`: Metadata formatting helper
- **ActionExecutor**: `case action_atom` timeout override — `:adjust_budget` → `:infinity` (prevents 5s Task.yield killing slow budget ops)
- **Router v32.0**: @always_sync_actions canonical 13-action list + timeout override mechanism

**Feb 17, 2026 - Adjust Budget Timeout Fix (WorkGroupID: fix-20260217-adjust-budget-timeout)**:
- **AdjustBudget v3.0 (75 lines)**: Cast-based budget update, unified code path
- Deleted Path A (adjust_child_directly, do_adjust, validate_adjustment)
Expand Down
142 changes: 100 additions & 42 deletions lib/quoracle/actions/dismiss_child.ex
Original file line number Diff line number Diff line change
Expand Up @@ -142,17 +142,19 @@ defmodule Quoracle.Actions.DismissChild do

# Snapshot budget state BEFORE termination (TreeTerminator v2.0 deletes cost records)
child_budget_data = get_child_budget_data(child_id, deps.registry)
tree_spent = query_child_tree_spent(child_id)
{tree_spent, per_model} = query_child_tree_by_model(child_id)

TreeTerminator.terminate_tree(child_id, parent_id, reason, deps)

# Reconcile budget: create absorption record, release escrow (v4.0)
# Reconcile budget: create per-model absorption records, release escrow (v5.0)
reconcile_child_budget(
parent_id,
parent_pid,
child_budget_data,
child_id,
task_id,
tree_spent,
per_model,
deps.pubsub
)

Expand Down Expand Up @@ -189,41 +191,47 @@ defmodule Quoracle.Actions.DismissChild do
end
end

# Query total tree spent before termination deletes records (v4.0)
@spec query_child_tree_spent(String.t()) :: Decimal.t()
defp query_child_tree_spent(child_id) do
own_costs = Aggregator.by_agent(child_id)
children_costs = Aggregator.by_agent_children(child_id)
# Query per-model tree costs before termination deletes records (v5.0)
# Returns {total_tree_spent, per_model_breakdown} for absorption record creation.
@spec query_child_tree_by_model(String.t()) :: {Decimal.t(), [map()]}
defp query_child_tree_by_model(child_id) do
per_model = Aggregator.by_agent_tree_and_model_detailed(child_id)

own_spent = own_costs.total_cost || Decimal.new("0")
children_spent = children_costs.total_cost || Decimal.new("0")
Decimal.add(own_spent, children_spent)
tree_spent =
Enum.reduce(per_model, Decimal.new("0"), fn row, acc ->
if row.total_cost, do: Decimal.add(acc, row.total_cost), else: acc
end)

{tree_spent, per_model}
end

# Reconcile budget with pre-queried spent (after tree termination) (v4.0)
# v4.1: Always create absorption record for non-zero costs to preserve task-level totals.
# Previously, N/A children had their cost records deleted by TreeTerminator with no
# replacement, causing task costs to drop.
# Reconcile budget with per-model breakdown (after tree termination) (v5.0)
# Creates per-model absorption records to preserve model attribution in Cost Details UI.
# Uses parent_id string directly (no Core.get_state call needed).
@spec reconcile_child_budget(
String.t(),
pid() | nil,
map() | nil,
String.t(),
binary() | nil,
Decimal.t(),
[map()],
atom() | nil
) :: :ok
defp reconcile_child_budget(
parent_id,
parent_pid,
child_budget_data,
child_id,
task_id,
tree_spent,
per_model,
pubsub
) do
# Step 1: Create absorption record for any non-zero costs (all budget modes)
create_absorption_record(parent_pid, child_budget_data, child_id, task_id, tree_spent, pubsub)

# Step 2: Release escrow only for allocated children
# Step 1: Release escrow FIRST for allocated children (requires live parent).
# This is the volatile operation — if parent crashes between escrow release and
# absorption, the committed budget is already freed (no leak). Absorption records
# are durable DB inserts that can be retried or created even with a dead parent.
case child_budget_data do
%{mode: :allocated, allocated: allocated} when not is_nil(allocated) ->
if parent_pid && Process.alive?(parent_pid) do
Expand All @@ -238,59 +246,109 @@ defmodule Quoracle.Actions.DismissChild do
:ok
end

# Step 2: Create per-model absorption records (no process liveness needed)
create_absorption_records(
parent_id,
child_budget_data,
child_id,
task_id,
tree_spent,
per_model,
pubsub
)

:ok
end

# Create absorption cost record under parent to preserve task-level totals.
# TreeTerminator deletes the child's original cost records, so this record
# ensures those costs remain visible in task-level aggregation queries.
@spec create_absorption_record(
pid() | nil,
# Create per-model absorption cost records under parent to preserve task-level totals
# and model attribution. TreeTerminator deletes the child's original cost records,
# so these records ensure costs remain visible in both task-level and per-model queries.
# Uses parent_id string directly — no process liveness check or Core.get_state needed.
@spec create_absorption_records(
String.t(),
map() | nil,
String.t(),
binary() | nil,
Decimal.t(),
[map()],
atom() | nil
) :: :ok
defp create_absorption_record(nil, _budget_data, _child_id, _task_id, _tree_spent, _pubsub) do
defp create_absorption_records(
_parent_id,
_budget_data,
_child_id,
_task_id,
_tree_spent,
[],
_pubsub
) do
:ok
end

defp create_absorption_record(parent_pid, budget_data, child_id, task_id, tree_spent, pubsub) do
if !Decimal.equal?(tree_spent, 0) && Process.alive?(parent_pid) do
try do
{:ok, parent_state} = Core.get_state(parent_pid)
parent_agent_id = parent_state.agent_id
defp create_absorption_records(
parent_id,
budget_data,
child_id,
task_id,
tree_spent,
per_model,
pubsub
) do
allocated = if budget_data, do: budget_data[:allocated], else: nil

allocated = if budget_data, do: budget_data[:allocated], else: nil
Enum.each(per_model, fn model_row ->
cost_usd = model_row.total_cost

if cost_usd && !Decimal.equal?(cost_usd, 0) do
cost_data = %{
agent_id: parent_agent_id,
agent_id: parent_id,
task_id: task_id,
cost_type: "child_budget_absorbed",
cost_usd: tree_spent,
metadata: %{
child_agent_id: child_id,
child_allocated: format_allocated(allocated),
child_tree_spent: decimal_to_string(tree_spent),
unspent_returned: format_unspent(allocated, tree_spent),
dismissed_at: DateTime.to_iso8601(DateTime.utc_now())
}
cost_usd: cost_usd,
metadata: build_absorption_metadata(model_row, child_id, allocated, tree_spent)
}

if pubsub do
Recorder.record(cost_data, pubsub: pubsub)
else
Recorder.record_silent(cost_data)
end
catch
:exit, _ -> :ok
end
end
end)

:ok
end

# Build absorption metadata for a per-model record.
# Includes model_spec and token data when present (LLM costs),
# omits model_spec for external/non-model costs.
# Preserves child_tree_spent and unspent_returned for backward compatibility.
@spec build_absorption_metadata(map(), String.t(), Decimal.t() | nil, Decimal.t()) :: map()
defp build_absorption_metadata(model_row, child_id, allocated, tree_spent) do
base = %{
"child_agent_id" => child_id,
"child_allocated" => format_allocated(allocated),
"child_tree_spent" => decimal_to_string(tree_spent),
"unspent_returned" => format_unspent(allocated, tree_spent),
"dismissed_at" => DateTime.to_iso8601(DateTime.utc_now())
}

if model_row.model_spec do
Map.merge(base, %{
"model_spec" => model_row.model_spec,
"input_tokens" => to_string(model_row.input_tokens || 0),
"output_tokens" => to_string(model_row.output_tokens || 0),
"reasoning_tokens" => to_string(model_row.reasoning_tokens || 0),
"cached_tokens" => to_string(model_row.cached_tokens || 0),
"cache_creation_tokens" => to_string(model_row.cache_creation_tokens || 0),
"input_cost" => decimal_to_string(model_row.input_cost || Decimal.new("0")),
"output_cost" => decimal_to_string(model_row.output_cost || Decimal.new("0"))
})
else
base
end
end

# Format allocated for absorption metadata
@spec format_allocated(Decimal.t() | nil) :: String.t()
defp format_allocated(nil), do: "N/A"
Expand Down
8 changes: 7 additions & 1 deletion lib/quoracle/actions/router.ex
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,14 @@ defmodule Quoracle.Actions.Router do
)

# For :call_mcp, fetch existing mcp_client from agent state (if any) before lazy-init
# Defense layer 3: liveness guard catches stale PID if Core hasn't processed DOWN yet
existing_mcp_client =
if action_type == :call_mcp, do: GenServer.call(agent_pid, :get_mcp_client), else: nil
if action_type == :call_mcp do
pid = GenServer.call(agent_pid, :get_mcp_client)
if pid && Process.alive?(pid), do: pid, else: nil
else
nil
end

# Build opts with agent metadata, then lazy-init MCP client only for :call_mcp
base_opts =
Expand Down
6 changes: 3 additions & 3 deletions lib/quoracle/actions/router/AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# lib/quoracle/actions/router/

## Sub-modules (extracted for 500-line limit, Router now 496 lines)
- MCPHelpers: MCP client lazy initialization (52 lines) - Added 2025-12-16
- MCPHelpers: MCP client lazy initialization (60 lines, v32.0: Process.alive? guard for stale PID detection) - Added 2025-12-16
- MockExecution: Mock action execution (174 lines)
- WaitFlow: Wait flow control (48 lines, v26.0 simplified to no-ops - all triggers moved to Agent layer)
- Metrics: Action metrics tracking (50 lines)
- TaskManager: Async task lifecycle (96 lines)
- Execution: Smart mode execution (264 lines, v31.0: Logger.warning on timeout)
- Execution: Smart mode execution (265 lines, v31.0: Logger.warning on timeout, v32.0: Logger.error in catch :exit block)
- ActionMapper: Action type→module mapping (29 lines)
- ShellCommandManager: Shell command state management (142 lines) - Added 2025-10-14, updated 2025-10-17
- Persistence: DB persistence for action results (135 lines) - Added 2025-10-14
Expand All @@ -16,7 +16,7 @@
- Security: Secret resolution and output scrubbing (added with secret system)

## Key Functions
- MCPHelpers.get_or_init_mcp_client/1: Get existing or create new MCP client from opts
- MCPHelpers.get_or_init_mcp_client/1: Get existing or create new MCP client from opts (v32.0: Process.alive? guard re-initializes dead clients)
- MCPHelpers.maybe_lazy_init_mcp_client/2: Only initializes for :call_mcp action type
- MockExecution.execute_action_mock/5: Simulates action responses
- WaitFlow.handle_immediate/3: Timer notification only for timed waits (no triggers - v26.0)
Expand Down
1 change: 1 addition & 0 deletions lib/quoracle/actions/router/execution.ex
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ defmodule Quoracle.Actions.Router.Execution do
{:error, {:action_crashed, Exception.message(e)}}
catch
:exit, reason ->
Logger.error("Action execution crashed (exit): #{inspect(reason)}")
{:error, {:action_crashed, reason}}
end
end)
Expand Down
6 changes: 4 additions & 2 deletions lib/quoracle/actions/router/mcp_helpers.ex
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,13 @@ defmodule Quoracle.Actions.Router.MCPHelpers do

require Logger

needs_init = (is_nil(mcp_client) or not Process.alive?(mcp_client)) and is_pid(agent_pid)

Logger.debug(
"MCPHelpers: mcp_client=#{inspect(mcp_client)}, agent_pid=#{inspect(agent_pid)}, creating_new=#{is_nil(mcp_client) and is_pid(agent_pid)}"
"MCPHelpers: mcp_client=#{inspect(mcp_client)}, agent_pid=#{inspect(agent_pid)}, creating_new=#{needs_init}"
)

if is_nil(mcp_client) and is_pid(agent_pid) do
if needs_init do
agent_id = Keyword.get(opts, :agent_id) || GenServer.call(agent_pid, :get_agent_id)

mcp_opts =
Expand Down
3 changes: 1 addition & 2 deletions lib/quoracle/actions/send_message.ex
Original file line number Diff line number Diff line change
Expand Up @@ -181,13 +181,12 @@ defmodule Quoracle.Actions.SendMessage do
# Look up agent's composite value to get parent info
case Registry.lookup(registry, {:agent, agent_id}) do
[{_pid, composite}] when is_map(composite) ->
parent_pid = Map.get(composite, :parent_pid)
parent_id = Map.get(composite, :parent_id)

# Has parent - look up parent to verify it exists
if parent_id do
case Registry.lookup(registry, {:agent, parent_id}) do
[{^parent_pid, _}] -> {:ok, [{:agent, parent_pid, parent_id}]}
[{pid, _}] -> {:ok, [{:agent, pid, parent_id}]}
_ -> {:error, :parent_not_found}
end
else
Expand Down
2 changes: 1 addition & 1 deletion lib/quoracle/actions/spawn.ex
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ defmodule Quoracle.Actions.Spawn do
{:error,
"Budget is required when spawning children. " <>
"Specify a budget amount (e.g., budget: \"50.00\"). " <>
"Use get_budget to check your available funds."}
"Check your <budget> context for available funds."}

error ->
error
Expand Down
2 changes: 1 addition & 1 deletion lib/quoracle/actions/spawn/budget_validation.ex
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ defmodule Quoracle.Actions.Spawn.BudgetValidation do
# N/A, nil, or unknown parent - child gets N/A budget (unlimited)
{:ok,
%{
child_budget_data: %{mode: :na, allocated: nil, committed: nil},
child_budget_data: Quoracle.Budget.Schema.new_na(),
escrow_amount: nil
}}
end
Expand Down
Loading