Skip to content

Add F1 radio RAG demo (full Opik eval loop)#12

Open
fschlz wants to merge 1 commit into
mainfrom
fschlz/feature/f1-radio-rag
Open

Add F1 radio RAG demo (full Opik eval loop)#12
fschlz wants to merge 1 commit into
mainfrom
fschlz/feature/f1-radio-rag

Conversation

@fschlz

@fschlz fschlz commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

What

A new use-cases/f1_radio_rag/ demo: a small, runnable Typer CLI that walks the entire Opik evaluation-and-improvement loop for a RAG use case — summarising F1 team-radio messages across a race weekend.

The loop (one CLI command each)

  • ingest — load radio messages into a local ChromaDB store (offline)
  • ask — retrieve + summarise with Claude (via litellm), traced in Opik
  • eval — build an Opik dataset + test suite; run plain-English assertions (run_tests) and the ContextRecall + Hallucination metrics (evaluate)
  • optimizeMetaPromptOptimizer improves the summariser prompt against the dataset
  • promote — save the optimised prompt to the Prompt Library (versioned)
  • run-all — the whole chain

Notes

  • uv project (pyproject.toml + uv.lock); pip install line also in the README.
  • Every command has a DRY_RUN path — runs with no credentials and prints useful output.
  • Credentials read from env vars only; .env.example provided.
  • Radio data is synthetic (OpenF1 team_radio is audio-only); the loop is identical for real transcripts. README carries this + the optimizer-scope caveat.
  • Indexed in use-cases/README.md.

use-cases/f1_radio_rag: a runnable Typer CLI walking the entire Opik loop
over a synthetic F1 team-radio RAG (ChromaDB): ingest -> ask (traced) ->
eval (dataset + test suite + ContextRecall/Hallucination metrics) ->
optimize (Optimization Studio) -> promote (Prompt Library). Every command
has a DRY_RUN path; credentials read from env vars only.
@fschlz fschlz self-assigned this Jun 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant