Diagnosis Engine

Diagnosis Engine

The diagnosis engine (engine/diagnose.py) drives the LLM through a bounded multi-turn loop, executes the proposed remediation against the cluster, and emits a structured result JSON.

DiagnosisEngine

DiagnosisEngine(
    api_key, model,
    namespace="default",
    max_turns=3,
    temperature=0.0,
    max_tokens=4096,
    base_url="https://openrouter.ai/api/v1",
    use_tools=True,
    max_retries=10,
    request_timeout=240,
    profile=None,
)

Built on the openai SDK so any OpenAI-compatible endpoint works (OpenRouter, Chutes, vLLM, etc.). The engine state is reset at the start of every diagnose() call.

Multi-Turn Loop

diagnose(scenario_description) runs up to max_turns turns:

1. Turn 1 sends the scenario description as the user message.
2. Subsequent turns send: "Based on the information gathered so far, please provide your final diagnosis and remediation steps as JSON."
3. If the response contains tool_calls, the engine executes each, appends the result as a role: "tool" message, and continues.
4. If the response contains text, the engine attempts _extract_json(content) — first as raw JSON, then json fenced, then fenced, then any {...} containing root_cause and remediation_steps, and finally a sliding JSONDecoder.raw_decode from each {.
5. A valid JSON object with a remediation_steps field terminates the loop.
6. If the loop exits without a parse, the engine sends one final prompt: "You have used all your investigation turns ... provide your final diagnosis and remediation steps as JSON now."

System Prompt

Built by _build_system_prompt(use_tools, profile). Two templates exist (with/without tools); both are filled with deployment_context (= profile.name) and context (= profile.context_prompt).

The tools template appends the rendered list of available function-calling tools and these rules:

• Always investigate first before proposing a fix.
• Remediation steps must be concrete kubectl commands with no placeholders.
• Do not include diagnostic commands (get, describe, logs) as remediation steps.
• Do not use interactive commands: edit, exec, watch, port-forward.
• For ConfigMap/Deployment fixes use kubectl patch, delete + apply, or set.
• Fix the root cause, not the symptoms.

The expected final answer schema:

{
  "root_cause": "one-line description",
  "reasoning": "investigation explanation",
  "remediation_steps": ["kubectl cmd 1", "kubectl cmd 2"]
}

Retry & Timeout Behavior

• API calls retry up to max_retries (default 10) on any Exception.
• Backoff is min(2**attempt + uniform(0, 1), 10) seconds.
• Each request also has timeout=request_timeout (default 240s, configurable via LLM_REQUEST_TIMEOUT).
• Token usage (prompt_tokens / completion_tokens) is accumulated for cost analysis.
• Every API failure is recorded in api_errors with attempt number, message, and ISO timestamp.

Output Shape

diagnose() returns:

{
  "diagnosis": {
    "root_cause": "...",
    "reasoning": "...",
    "remediation_steps": [...],
    "turns_used": <int>
  },
  "session_meta": {
    "model", "namespace", "use_tools", "max_turns", "temperature",
    "duration_s", "turns_used", "timestamp",
    "total_input_tokens", "total_output_tokens", "total_tokens"
  },
  "turn_timings": [{"turn", "duration_s", "action", "tools_called", "input_tokens", "output_tokens"}],
  "api_errors": [{"attempt", "error", "timestamp"}],
  "has_actionable_remediation": <bool>,
  "conversation_history": [...]  // full chat including tool calls/results
}

has_actionable_remediation filters out diagnostic / blocked / dangerous commands; it is True iff at least one real fix command was proposed.

Remediation Execution

execute_remediation(result, namespace) walks remediation_steps and:

• Marks blocked_dangerous (regex match against _DANGEROUS_PATTERNS).
• Marks skipped_diagnostic (get / describe / logs / ...).
• Marks blocked (interactive subcommand: edit / exec / port-forward / watch / proxy / attach).
• Forces a -n <namespace> argument; commands targeting a different namespace are rejected outright.
• Runs surviving commands with shell=False and timeout=REMEDIATION_STEP_TIMEOUT (default 60s).

See Safety & Guardrails for the full filter list.

CLI

engine/diagnose.py is runnable directly:

python3 -m engine.diagnose \
    --model z-ai/glm-5 \
    --profile open5gs \
    --scenario-description "..." \
    --max-turns 3 \
    --output diagnosis.json

Useful for replaying diagnosis without the full shell pipeline.