[Bug] Dialectic "minimal" reasoning level returns null on tool-less empty responses

[leo922oel]

🐞 Bug Report

Describe the bug

At reasoning_level="minimal", peer.chat(...) returns null when the model answers without calling a tool (common for broad queries like "summarize what you know about X"). The same query at low or medium returns a normal answer from the same stored data.

To be clear about scope: the minimal tier's low-cost profile looks intentional and is not the concern here. It uses the same model as low but with MAX_TOOL_ITERATIONS=1, MAX_OUTPUT_TOKENS=250, a reduced toolset, and smaller prefetch. That is a reasonable cheap, single-shot tier.

The concern is that an empty completion is returned as null with no recovery, even though execute_tool_loop (src/llm/tool_loop.py) already has an empty-response retry meant to prevent exactly this. The retry is guarded by iteration < max_tool_iterations - 1. With MAX_TOOL_ITERATIONS=1 that is 0 < 0, which is False, so the retry never fires at minimal. The tool-less branch then returns before reaching the post-loop synthesis fallback (that fallback is only reached after iterations are exhausted through tool calls). low and medium avoid this only because they allow MAX_TOOL_ITERATIONS >= 2.

Is the null intended for minimal, or should the empty-response recovery cover the single-iteration case? The existing retry suggests empty responses are meant to be recovered, and the exclusion of minimal reads like a side effect of the guard arithmetic rather than a deliberate choice — the tier (#459) and the guard (#814) came from separate changes. Filing to confirm the intent; if it is a bug, I have a fix and a regression test ready.

One more data point: the null response used ~24 output tokens (well under the 250 cap), so this is an empty completion with no fallback, not a truncation at the token limit.

---

Is this a regression?

The empty-response retry was added in #814. The minimal tier (MAX_TOOL_ITERATIONS=1) predates it (#459). The guard has excluded the single-iteration case since #814.

---

To Reproduce

Deterministic unit test, no LLM or network needed. Add to tests/llm/ and run on a clean main:

from typing import Any
import pytest
from src.llm import tool_loop
from src.llm.runtime import AttemptPlan
from src.llm.tool_loop import execute_tool_loop
from src.llm.types import HonchoLLMCallResponse


def _plan() -> AttemptPlan:
    return AttemptPlan(
        provider="anthropic", model="claude-sonnet-4-5", client=object(),
        thinking_budget_tokens=None, reasoning_effort=None, selected_config=None,
        attempt=1, retry_attempts=1, is_fallback=False,
    )


def _resp(content: str) -> HonchoLLMCallResponse[Any]:
    return HonchoLLMCallResponse(
        content=content, input_tokens=10, output_tokens=5,
        cache_creation_input_tokens=0, cache_read_input_tokens=0,
        finish_reasons=["stop"], tool_calls_made=[],
    )


class _EmptyThenRecover:
    def __init__(self) -> None:
        self.tool_calls = 0

    async def __call__(self, *_a: Any, **kw: Any) -> HonchoLLMCallResponse[Any]:
        if kw.get("tools") is None:  # forced synthesis call
            return _resp("recovered via synthesis")
        self.tool_calls += 1
        return _resp("" if self.tool_calls == 1 else "recovered via retry")


async def _run(max_iters: int, mock: _EmptyThenRecover):
    return await execute_tool_loop(
        prompt="hi", max_tokens=250,
        messages=[{"role": "user", "content": "What do you know about me?"}],
        tools=[{"name": "noop", "description": "no-op", "input_schema": {"type": "object"}}],
        tool_choice="auto", tool_executor=lambda _n, _i: "",
        max_tool_iterations=max_iters, response_model=None, json_mode=False,
        temperature=None, stop_seqs=None, verbosity=None, enable_retry=False,
        retry_attempts=1, max_input_tokens=None, get_attempt_plan=_plan,
        before_retry_callback=lambda _r: None, stream_final=False, telemetry=None,
    )


@pytest.mark.asyncio
async def test_minimal_single_iteration_returns_null():   # fails on main
    mock = _EmptyThenRecover()
    with pytest.MonkeyPatch.context() as mp:
        mp.setattr(tool_loop, "honcho_llm_call_inner", mock)
        result = await _run(1, mock)
    assert result.content and result.content.strip(), f"got {result.content!r}"


@pytest.mark.asyncio
async def test_two_iterations_recover():                  # passes on main
    mock = _EmptyThenRecover()
    with pytest.MonkeyPatch.context() as mp:
        mp.setattr(tool_loop, "honcho_llm_call_inner", mock)
        result = await _run(2, mock)
    assert result.content == "recovered via retry"

Steps:

1. Add the file above to tests/llm/.
2. Run uv run pytest tests/llm/<file>.py -v.
3. test_minimal_single_iteration_returns_null fails with result.content == ''. The max_tool_iterations=2 control passes.

Behavioral path: call peer.chat(query, reasoning_level="minimal") with a broad query the model answers without a tool call. The response is null. The same query at low returns an answer.

---

Expected behaviour

An empty tool-less response should be recovered rather than returned as null: fire the existing retry when iterations remain, and fall through to the forced synthesis call when they do not (the minimal case). minimal can stay single-shot and cheap; it just should not drop the answer. A grounded "I don't have information about X" is an acceptable result; bare null is not.

---

Your environment

• OS: Ubuntu (Linux 6.8 x86_64)
• Honcho Server Version: reproduced on main @ 4f9a4136
• Honcho Client Version: N/A (server-side tool-loop logic)

---

Additional context

The guarded block in src/llm/tool_loop.py:

if not response.tool_calls_made:
    if (
        isinstance(response.content, str)
        and not response.content.strip()
        and empty_response_retries < 1
        and iteration < max_tool_iterations - 1   # 0 < 0 == False when MAX_TOOL_ITERATIONS=1
    ):
        empty_response_retries += 1
        conversation_messages.append({"role": "user", "content": "...provide a concise answer..."})
        iteration += 1
        continue
    # returns `response` here; the post-loop synthesis call is never reached

minimal's MAX_TOOL_ITERATIONS=1 is set in src/config.py (_default_dialectic_levels).

Searched open and closed issues; the closest are related but do not cover this:

• [Bug] Strict OpenAI-compatible models (Mistral) fail with 'Unexpected role user after role tool' when max_tool_iterations is exhausted #747 touches the same retry/synthesis path from a different angle (appending a role: user message after a role: tool message breaks strict OpenAI-compatible models). A fix here should keep that in mind.
• [Bug] AsyncAnthropicVertex returns empty content in tool-calling loops (deriver/dream) #536 is the same symptom (empty content in the tool loop) with a different cause (provider returning None).