Problem / Motivation
As the bot evolves into a full-fledged ML-driven trading system, the need for a standardized backtest engine looks quite critical.
Currently, the backtesting logic is implemented through ad-hoc scripts (e.g., simulate_model) and custom performance functions (simulated_trade_performance). While functional, this approach lacks:
- Consistency across different strategies and configurations
- Reusability for rapid experimentation and validation
- Support for live vs simulated parity, a must for bridging to live trading
- Extensibility to support risk metrics, multiple assets, slippage, commission, and model retraining over time
For ML-based systems, where models are trained on past data and evaluated in a forward-looking manner (e.g., walk-forward), a formal backtesting engine ensures fair, repeatable, and accurate performance estimation.
PS: I think we can use VectorBT as the backend for this.
Objective
Design and implement a modular, strategy-agnostic Backtest Engine that:
- Accepts configurable parameters (thresholds, signal columns, etc.)
- Supports rolling retraining / evaluation of ML models
- Evaluates strategies using both price and derived ML signals
- Produces detailed metrics (profit, Sharpe, win-rate, drawdown, etc.)
- Can be extended to support VectorBT, CCXT, and live execution parity
Structure
-
Backtest Inputs:
DataFrame with OHLCV and model predictions- Strategy config (buy/sell thresholds, retrain frequency, etc.)
- Evaluation period split (train/test windows)
-
Backtest Outputs:
- Trade logs
- Summary metrics
Evaluation Metrics to Include
- Net profit / % profit
- Sharpe ratio / Sortino ratio
- Max drawdown
- Number of trades
- Win rate
- Profit factor
- Profit per month / trade
📌 Tasks
If anyone has prior experience with backtesting engines (e.g., Zipline, Backtrader, VectorBT, bt), feel free to share thoughts, patterns, or contribute prototypes. Design suggestions are welcome!
Problem / Motivation
As the bot evolves into a full-fledged ML-driven trading system, the need for a standardized backtest engine looks quite critical.
Currently, the backtesting logic is implemented through ad-hoc scripts (e.g.,
simulate_model) and custom performance functions (simulated_trade_performance). While functional, this approach lacks:For ML-based systems, where models are trained on past data and evaluated in a forward-looking manner (e.g., walk-forward), a formal backtesting engine ensures fair, repeatable, and accurate performance estimation.
PS: I think we can use VectorBT as the backend for this.
Objective
Design and implement a modular, strategy-agnostic Backtest Engine that:
Structure
Backtest Inputs:
DataFramewith OHLCV and model predictionsBacktest Outputs:
Evaluation Metrics to Include
📌 Tasks
BacktestEngineand interfacesimulated_trade_performance()into engine-compatible moduleThresholdStrategyandMLBasedStrategyvectorbtorbtas backendREADME.mdIf anyone has prior experience with backtesting engines (e.g., Zipline, Backtrader, VectorBT, bt), feel free to share thoughts, patterns, or contribute prototypes. Design suggestions are welcome!