Architecture

Architecture

OperAID is organized around a small set of orthogonal concerns: an LLM diagnosis engine, a pluggable tool registry, a deployment profile that abstracts the workload-under-test, and shell-level runners that wire fault injection and verification around each LLM session.

Pipeline

Each experiment run executes the following pipeline:

1. Pre-flight health check — first run in a suite performs a full nuke for a clean baseline. Subsequent runs skip the nuke if the cluster is already healthy; the previous run's LLM diagnosis phase provides a natural stabilization window.
2. Fault injection — deterministic fault applied via the profile-defined method (kubectl_apply / kubectl_patch / kubectl_scale).
3. Agentic diagnosis — multi-turn LLM reasoning with function-calling access to kubectl_* diagnostic tools (configurable).
4. Remediation — proposed kubectl commands executed against the cluster, after passing safety filters.
5. Execution-based verification — health checks confirm all expected deployments are ready.
6. Fallback recovery — if LLM remediation fails, a fast reset (helm upgrade) is attempted, with full nuke as last resort.

Repository Layout

operaid/
├── config.env                 # Main configuration (LLM, K8s, timeouts)
├── run_experiment.sh          # Single experiment runner (~573 lines of bash)
├── run_suite.py               # Suite runner with YAML config
├── regenerate_stats.py        # Rebuild suite_statistics.json + summary.csv
├── nuke-deployment.sh         # Full cleanup / redeploy
├── reset-deployment.sh        # Fast reset via helm upgrade
├── visualize.py               # Single-run visualizer
├── visualize_suite.py         # Suite visualizer (paper figures)
├── scenario_definitions.json  # Legacy scenario doc (profiles are authoritative)
├── deployments/
│   └── open5gs.json           # Deployment profile (faults, health, tools, prompts)
├── engine/
│   ├── diagnose.py            # Multi-turn LLM diagnosis engine
│   └── profile.py             # Deployment-profile loader and name resolver
├── tools/
│   └── kubectl_tools.py       # Built-in kubectl tools + custom-tool registry
├── health/
│   └── health_check.py        # Generic deployment readiness check
├── scenarios/                 # YAML manifests applied by fault injectors
├── suites/                    # YAML suite definitions (experiment matrices)
├── paper/                     # LaTeX source, figures, tables
└── suite_results/             # Per-run outputs (git-ignored)

Component Map

                ┌─────────────────────────────┐
                │      run_suite.py           │
                │  (matrix expansion, CSV     │
                │   aggregation, stats)       │
                └──────────────┬──────────────┘
                               │ invokes per run
                               ▼
                ┌─────────────────────────────┐
                │   run_experiment.sh         │
                │   nuke / inject / verify    │
                └──────────────┬──────────────┘
                               │ calls
                               ▼
        ┌──────────────────────────────────────────────┐
        │     engine/diagnose.py                       │
        │     DiagnosisEngine — multi-turn loop        │
        │                                              │
        │     ┌──────────────────────────────────┐     │
        │     │ tools/kubectl_tools.py           │     │
        │     │  built-ins + profile customs     │     │
        │     └──────────────────────────────────┘     │
        │     ┌──────────────────────────────────┐     │
        │     │ engine/profile.py                │     │
        │     │  name resolution, context, faults│     │
        │     └──────────────────────────────────┘     │
        └──────────────────────────────────────────────┘
                               │
                               ▼
                ┌─────────────────────────────┐
                │ health/health_check.py      │
                │ deployments readiness gate  │
                └─────────────────────────────┘

Generalization via Profiles

OperAID does not hard-code Open5GS. Everything workload-specific — namespace, NF names, expected deployments, fault injection method, custom tools, the LLM context prompt — lives in deployments/<profile>.json. To add a new target (e.g. free5GC), drop in a new profile JSON; no code changes required. See Deployment Profiles.