Suite Configuration

Suite Configuration

Suites are YAML files in suites/ consumed by run_suite.py. They expand into a flat list of (experiment, scenario, run) tuples; the runner invokes run_experiment.sh for each.

Shipped Suites

File	Purpose
tool_impact.yaml	Headline experiment — 5 models × 2 conditions × 3 scenarios × 30 runs = 900
single_model_quick.yaml	Smoke test, one model, low run count
validation_test.yaml	Sanity check for new code paths
bugfix_validation.yaml	Regression matrix for fixes
temperature_sweep.yaml	Holds model/scenario fixed; varies temperature
scenario_deep_dive.yaml	Single scenario, many runs, multiple models

Schema

name: <suite-name>                 # used in output directory naming

common:                            # defaults merged into each experiment
  profile: open5gs
  scenarios: [1, 2, 3]
  runs_per_scenario: 30
  max_turns: 3
  temperature: 0.0
  max_tokens: 4096
  base_url: "https://openrouter.ai/api/v1"
  custom_tools: []

experiments:                       # list of variants
  - name: "glm-5-all-tools"
    model: "z-ai/glm-5"
    use_tools: true
  - name: "glm-5-no-tools"
    model: "z-ai/glm-5"
    use_tools: false

build_experiment_list() in run_suite.py merges each experiment with common (experiment wins on conflict), then materializes the cartesian product over scenarios × runs_per_scenario.

Field Reference

Field	Default	Notes
profile	open5gs	Deployment profile name or path.
scenarios	[1, 2, 3]	Scenario IDs. Strings like "1,2,3" are also accepted.
runs_per_scenario	CLI --runs (3)	Repetitions per scenario per experiment.
max_turns	3	Hard cap on LLM tool-use turns.
temperature	0.0	Passed straight to the OpenAI client.
max_tokens	4096	Per-response output cap.
base_url	OpenRouter	Override for Chutes / vLLM / local proxies.
model	required	Model identifier (provider-prefixed).
use_tools	true	When false, the engine never sends tools=....
custom_tools	[]	If non-empty, overrides the profile's custom_tools.
name	derived	Used in result paths; defaults to `-<all

Tool-Impact Matrix (April 2026)

The 900-run headline experiment used these 5 models, each in both conditions:

• z-ai/glm-5
• openai/gpt-oss-120b
• moonshotai/kimi-k2.5
• qwen/qwen3.5-35b-a3b
• qwen/qwen3.5-397b-a17b

Result: tool access raised average success from 7.1% → 70.7% (+63.6 pp). The small Qwen3.5-35b-a3b (3B active params) leads at 93.3% with tools.

Output Locations

Suite results land under:

suite_results/<suite-name>/<YYYYMMDD-HHMMSS>/
    <experiment-name>/
        run-1/, run-2/, ...
    summary.csv
    suite_statistics.json
    api_error_summary.csv

See Results & Outputs.