Core data preprocessing engine for DeepAlpha Terminal.
This repository contains the standardized ETL (Extract, Transform, Load) scripts used by Trading Brains Studio to process high-frequency futures data from Chinese exchanges (SHFE, DCE, CZCE).
- ⚡ High Performance: Optimized for processing large CSV tick data using Pandas vectorization.
- 🧹 Smart Cleaning: Auto-detection of bad ticks and outliers in OHLC streams.
- 🔄 Format Conversion: Converts raw exchange CSVs into TqSdk-compatible formats (HDF5/Parquet).
- 🛡️ Reliability: robust missing value handling for continuous contract construction.
from cleaner import FuturesDataCleaner
# Initialize pipeline
pipeline = FuturesDataCleaner("data/raw/rb_main.csv")
# Execute cleaning steps
pipeline.load_data()
pipeline.remove_outliers(threshold=0.05)
pipeline.export_processed_data("data/clean/rb_main.h5")
Integration
This module is integrated into the backend of DeepAlpha Pro.
© 2026 Trading Brains Studio.