k6 LLM Benchmark

Benchmark OpenAI-compatible LLM servers with k6 using a Wikipedia prompt dataset. This repo sends streaming chat completions to a /v1/chat/completions endpoint, captures token-level performance metrics, and provides a Python utility to aggregate results to CSV.

What this repo does

• Loads prompts from a Wikipedia dataset (JSONL with id and text).
• Sends streamed chat completion requests to an OpenAI-compatible server.
• Records metrics like first-token latency, prompt TPS, and completion TPS.
• Exports k6 summary JSON per run.
• Aggregates JSON results into a CSV for analysis.

Requirements

• Go (for building a custom k6 binary)
• k6 with SSE support (k6/x/sse)
• A running OpenAI-compatible LLM server
• Python 3 with pandas (for data aggregation)

k6 with SSE support

The test script imports k6/x/sse, which requires a custom k6 build via xk6 and an SSE extension (for example, github.com/phymbert/xk6-sse). Build k6 and set K6_EXECUTABLE accordingly.

Dataset format

The script expects a JSONL file with one object per line:

{"id": "...", "text": "..."}

The batch_test.sh file expects datasets like:

${WIKI_DATASET_DIR}/prompts_128.txt
${WIKI_DATASET_DIR}/prompts_256.txt

Update WIKI_DATASET_DIR in the script or pass SERVER_BENCH_DATASET to point at your dataset.

Quick start

1. Start your LLM server with OpenAI-compatible endpoints:

   • GET /v1/models
   • POST /v1/chat/completions (streaming SSE)

2. Run a single test:

SERVER_BENCH_URL=http://localhost:8080/v1 \
SERVER_BENCH_MODEL_ALIAS=my-model \
SERVER_BENCH_DATASET=/path/to/prompts_128.txt \
SERVER_BENCH_MAX_TOKENS=128 \
K6_EXECUTABLE=/path/to/k6 \
/path/to/k6 run script.js --vus 4 --iterations 128 --summary-export results/run.json

3. Run batch tests:

./batch_test.sh

The batch script sets defaults like:

• SERVER_BENCH_URL (server endpoint)
• WIKI_DATASET_DIR (dataset root)
• K6_EXECUTABLE (custom k6 path)
• concurrent_users, input_output_lens, rep_ids

Edit these values in batch_test.sh for your environment.

Environment variables

The k6 script reads the following:

• SERVER_BENCH_URL (default: http://localhost:8080/v1)
• SERVER_BENCH_MODEL_ALIAS (default: my-model)
• SERVER_BENCH_DATASET (default: ./ShareGPT_V3_unfiltered_cleaned_split.json)
• SERVER_BENCH_MAX_TOKENS (default: 512)
• SERVER_BENCH_N_PROMPTS (default: 600 / 10 * 8)

Output

k6 exports JSON summary files (one per run). The data collector flattens these into a CSV.

Collect results to CSV

python collect_data.py \
  --input_dir ./results/<your-run-dir> \
  --output_file ./results/summary.csv

This extracts fields like:

• llamacpp_emit_first_token_second
• llamacpp_prompt_processing_TPS
• llamacpp_tokens_TPS
• llamacpp_prompt_tokens
• llamacpp_completion_tokens
• iterations, iteration_duration, vus

Notes

• The request payload is sent via SSE streaming and includes stream_options.include_usage = true.
• The script randomizes word order in each prompt to reduce caching effects.
• Results are stored under the results/ directory; batch runs create per-run subfolders and copy the batch script for reproducibility.