Feat: .cpp comments, README.md, documentation & compare_runs.py

Author: ltanak

README.md

@@ -1,10 +1,74 @@
 # Evaluating the performance of lock-free data structures under realistic exchange workloads
-Louis Tanak, University of Warwick Third Year Computer Science Project
+Louis Tanak, University of Warwick Third Year Computer Science Project 2026.
 
 ## Abstract
-Lock-free data structures allow multiple threads to operate safely on shared data without using locks. These have many real-word use cases in high-performance computing, low-latency systems and financial exchanges. Research has been conducted on their theoretical properties, however there is a gap in understanding how lock-free data structures perform under realistic workloads, such as trading exchange-like patterns. 
+Lock-free data structures are concurrent data structures that allow multiple threads to operate without the use of locks for synchronisation. They utilise atomic hardware instructions such as compare-and-swap (\acrshort{CAS}) or fetch-and-add (\acrshort{FAA}) to guarantee that at least one thread makes progress in a finite number of steps. Due to their ability to reduce contention and improve scalability, lock-free data structures have various applications primarily in high-performance systems such as web servers, physics engines and high-frequency trading systems. 
 
-The work aims to implement various lock-free data structures and evaluate their performance against each other and traditional lock-based solutions. The data structures will be benchmarked against simulated exchange traffic to measure throughput, latency, scalability and completion ordering. Results will provide insights into the benefits, drawbacks and the suitable conditions for each data structure.
+Modern research covers different implementations of various lock-free data structures, to improve their latency and scalability with increasing thread counts. However, there currently lacks standardised frameworks to empirically evaluate lock-free data structures. Further to this, studies primarily focus on throughput and latency under synthetic workloads, whilst little work investigates order-preservation properties and their effects on downstream systems. These are critical properties in applications such as financial exchanges, where fairness and correctness depend on ordering guarantees.
+
+This project evaluates the performance of lock-free data structures under realistic exchange workloads. We present a low-latency benchmarking framework, which provides concurrent measurement of latencies, order-preservation and implements a price-time priority matching engine to assess the downstream effects of ordering anomalies. Using the framework, we present a performance analysis of various lock-free data structure implementations and their characteristics. We conclude that the framework can effectively evaluate performance and ordering properties, providing a reusable suite for evaluating lock-free data structures.
+
+## Supervisor Running Instructions
+This section is a quickstart intended for the project supervisor. It walks through running any one of the six lock-free data-structure implementations out of the box and viewing the generated report.
+
+### 1. Compile the project
+From the repository root, run:
+```bash
+./compile.sh
+```
+
+### 2. Select the data structure to evaluate
+Open `src/main.cpp`. Near the top of `main()` you will find six clearly-labelled blocks under the **DATA STRUCTURE SELECTION** banner:
+
+1. `RegularQueue` - lock-based baseline
+2. `MCConcurrentQueue` - moodycamel ConcurrentQueue (MPMC)
+3. `MCLockFreeQueue` - moodycamel ReaderWriterQueue (SPSC only)
+4. `WiltMPMCBlockRing` - Wilt MPMC blocking ring buffer
+5. `WiltMPMCNonBlockRing` - Wilt MPMC non-blocking ring buffer
+6. `RigtorpMPMCQueue` - Rigtorp MPMC queue *(active by default)*
+
+Uncomment **exactly one** block and comment out the others, then re-run `./compile.sh`.
+
+By default, `RigtorpMPMCQueue` is uncommented.
+
+### 3. Run the benchmark
+The recommended testing scenario runs both the stress and order scenarios under the same run ID:
+```bash
+./run.sh --all 4 1000
+```
+The arguments are:
+- `4` - number of producer/consumer threads (use `1` for SPSC-only structures such as the moodycamel ReaderWriterQueue)
+- `1000` - total orders driven through the system
+
+The script prints a run ID of the form `XXXXXXXXXX` at the start and end of the run. **Copy this run ID**, as it is needed for the report step.
+
+### 4. Generate the HTML report
+Switch into the Python benchmarking directory and generate the report for the run ID from step 3:
+```bash
+cd src/benchmarking/python
+pixi run report --run-id=XXXXXXXXXX
+```
+(Replace `XXXXXXXXXX` with the actual ID printed by `run.sh`.)
+
+The first invocation of `pixi run` will download the Python environment, with subsequent calls being faster once the environment is resolved.
+
+### 5. View the report
+The generated HTML report is written to:
+```
+results/reports/report_<run_id>.html
+```
+Open this file in any browser. It contains latency plots, ordering summaries, exchange-matching results and hardware counter data for the selected data structure.
+
+### Full supervisor walkthrough (copy-paste)
+```bash
+./compile.sh
+# edit src/main.cpp, uncomment the desired data structure, re-run compile.sh
+./run.sh --all 4 1000
+# note the run ID printed by run.sh
+cd src/benchmarking/python
+pixi run report --run-id=<RUN_ID>
+# open results/reports/report_<RUN_ID>.html in a browser
+```
 
 ## Adding your own Lock-free Data Structure
 To begin benchmarking your lock-free data structure, perform the following:
@@ -157,4 +221,10 @@ If you want direct plots instead of the report:
 
 If you would prefer to look at the code, please see: `pixi.toml` and `main.py` (both are unique names in this codebase!)
 
-Note: The plotting code is actively evolving. The report mode is the most stable path for benchmarking.
+### Cross-run latency comparison
+To overlay latency graphs from multiple runs onto a single image (useful for comparing different data structures), see `src/benchmarking/python/image_editing/compare_runs.py` or run:
+- `pixi run compare --run-ids 1234567 7654321 1122334`
+
+Output images are saved into `src/benchmarking/python/image_editing/`.
+
+Note: The plotting code is actively evolving. The report mode is the most stable path for benchmarking.

documentation/README.md

@@ -0,0 +1,16 @@
+# Framework Documentation
+
+This directory contains reference documentation for the lock-free data-structure benchmarking framework. It is split into two parts:
+
+- [`cpp.md`](cpp.md) - the C++ benchmarking core (data structures, scenarios, benchmarking harness, hardware logging)
+- [`python.md`](python.md) - the Python analysis pipeline (plotting, report generation, cross-run comparison, image editing)
+
+For a quickstart aimed at running the framework end-to-end, see the **Supervisor Instructions** section in the top-level [`README.md`](../README.md).
+
+## High-level architecture
+
+The framework is split across three phases: a C++ execution phase that drives producer/consumer threads through the selected data structure, a post-processing phase that emits one CSV per metric category, and a Python analysis phase that turns those CSVs into an HTML report.
+
+![System architecture](images/system_architecture.png)
+
+The C++ side produces tagged CSV files under `results/`. The Python side consumes those CSVs, groups them by `run_id`, and produces either per-plot images or a single aggregated HTML report.

documentation/cpp.md

@@ -0,0 +1,98 @@
+# C++ Framework
+
+The C++ side of the project does three things: it owns the data structures under test, it drives them with realistic-ish exchange traffic, and it dumps measurements to CSV for the Python pipeline to pick up. This document is a tour of how that's wired together.
+
+## Layout
+
+Headers live under `include/`, implementations under `src/`, and the two trees mirror each other:
+
+```
+include/
+  benchmarking/        BenchmarkWrapper template
+  data_structures/
+    queues/            IQueue interface + queue implementations
+    ring_buffers/      IRing interface + ring buffer implementations
+  exchange/            Matching engine, order book, price levels
+  hardware_logging/    PAPI counter wrappers
+  order_simulation/    Synthetic order generators
+  scenarios/           Stress / order scenario drivers and CLI parsing
+  utils/               Timing, threading, file and struct helpers
+
+src/
+  main.cpp                        Entry point and data structure selection
+  benchmarking/cpp/benchmark.cpp  Explicit BenchmarkWrapper instantiations
+  data_structures/                Queue / ring implementations
+  exchange/                       Matching engine
+  hardware_logging/               PAPI wrapper
+  order_simulation/               Order generators
+  scenarios/                      CLI parsing (test_inputs)
+  utils/                          Timing / threading / file helpers
+```
+
+## main.cpp
+
+`main.cpp` is intentionally small. It parses arguments into a `TestParams`, picks a ring-buffer capacity based on the workload, and constructs a `BenchmarkWrapper` around exactly one of the six candidate data structures. The six candidates are arranged as mutually-exclusive blocks under a "DATA STRUCTURE SELECTION" banner — to switch which one is benchmarked, comment the active block out and uncomment the one you want, then rebuild. Once the wrapper exists, `main` just dispatches into either the stress or the order scenario and exits.
+
+## BenchmarkWrapper
+
+`BenchmarkWrapper<TDS, TOrder>` (in `include/benchmarking/benchmark.hpp`) is the heart of the framework. It is templated on both the data structure type and the order type, so any class implementing `IQueue` or `IRing` slots in without modification.
+
+![BenchmarkWrapper class diagram](images/benchmark_wrapper_class.png)
+
+The wrapper handles everything around the hot path: it tracks producer and consumer thread IDs so per-thread metrics make sense, takes nanosecond timestamps either side of every enqueue and dequeue, attaches sequence numbers so the Python side can reason about FIFO ordering, runs orders through the matching engine when the order scenario asks it to, and wraps each measured region in start/stop calls to `HardwareLogger`. When a scenario finishes the wrapper writes one CSV per metric category into `results/`, tagged with the run ID and timestamp that `run.sh` passed in.
+
+Because every supported data structure needs its own template instantiation, those instantiations are collected at the bottom of `src/benchmarking/cpp/benchmark.cpp`. Adding a new data structure means adding one more `template class BenchmarkWrapper<...>;` line there.
+
+## Data structures
+
+There are two interfaces, `IQueue<TOrder, Derived>` for queue-style structures and `IRing<TOrder, Derived>` for bounded ring buffers. Both are CRTP, so calls from the wrapper into the data structure are statically dispatched and the harness contributes no virtual-call overhead to the measurements. The contract for both interfaces is the same five operations:
+
+```cpp
+auto enqueueOrder(TOrder &order) -> bool;
+auto dequeueOrder(TOrder &order) -> bool;
+auto getSize()                   -> uint64_t;
+auto isEmpty()                   -> bool;
+auto getFront(TOrder &order)     -> bool;
+```
+
+The recipe for adding a new data structure lives in the top-level README under "Adding your own Lock-free Data Structure".
+
+## Scenarios
+
+There are two scenarios, `stress` and `order`, defined under `include/scenarios/`. Stress just hammers the data structure with as many enqueue/dequeue operations as the configured thread count and order count permit; it produces latency and hardware CSVs. Order is the more interesting one — it drives a mixed workload through the matching engine, checks per-item ordering, and emits ordering, exchange and hardware CSVs.
+
+CLI parsing lives in `scenarios/test_inputs.hpp`. It populates a `TestParams` from `argv`, including the positional `mode / threads / orders` arguments and the optional `--seed` and `--run-id` flags that `run.sh` always supplies.
+
+## Order simulation
+
+The synthetic order flow lives under `include/order_simulation/`. Each generator implements `IOrderGenerator`, and `collection_order_generator` multiplexes across several sub-generators (random, momentum, mean-reverting, market maker) using a shared `MarketState` so the resulting traffic looks roughly like real exchange flow rather than uniform noise. `BenchmarkOrder` is the POD passed through the data structures; everything in the C++ side is templated over it.
+
+## Exchange
+
+`include/exchange/` contains a small limit-order-book matching engine. It is deliberately not a full exchange — it just consumes orders from the data structure, updates the price book and per-level state, and emits a trades cycle that the wrapper writes out to the exchange CSV. The Python side then compares those trades against the expected trades for the input sequence.
+
+## Hardware logging
+
+`HardwareLogger` is a thin wrapper over PAPI for collecting cycles, instructions retired, L1/L2 cache misses and branch mispredictions. `thread_counter` keeps the counters per-thread, which is what lets the hardware CSV distinguish producer behaviour from consumer behaviour. `hardware_metrics` defines the schema that ends up as the CSV header.
+
+## Utilities
+
+The `utils/` directory is where the smaller helpers live. `timing.hpp` exposes nanosecond clocks and RDTSC; `threads.hpp` handles thread pinning; `files.hpp` knows how to build the `<category>_<run_id>_<timestamp>.csv` filenames the Python side expects; `structs.hpp` holds shared PODs; `counter.hpp` has a couple of small atomic counters.
+
+## Build
+
+Building is one command via CMake + Ninja:
+
+```bash
+./compile.sh
+```
+
+The result is a single executable at `build/lock_free_data_structures` which `run.sh` invokes with the appropriate flags.
+
+## End-to-end flow
+
+From a single `./run.sh` invocation, here is what happens:
+
+![Process flow](images/process_flow.png)
+
+`run.sh` generates a random run ID and launches the executable with the parsed arguments and `--run-id <RUN_ID>`. `main.cpp` constructs the chosen data structure plus a wrapper around it, then hands control to the stress or order scenario. The scenario drives the wrapper; the wrapper accumulates latency, ordering and hardware counter samples in memory. When the scenario finishes the wrapper writes one CSV per category into the right subdirectory of `results/`. From there the Python pipeline (covered in [`python.md`](python.md)) picks the CSVs up by run ID and turns them into plots or an aggregated HTML report.