Add FleetSim workload forecast backtests#2597
Conversation
Signed-off-by: haowu1234 <13258260125@163.com>
Signed-off-by: haowu1234 <13258260125@163.com>
✅ Deploy Preview for vllm-semantic-router ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
github-actions
Bot
assigned
AayushSaini101,
drivebyer,
FAUST-BENCHOU,
ramkrishs,
shraderdm and
WUKUNTAI-0211
👥 vLLM Semantic Team NotificationThe following members have been identified for the changed files in this PR and have been automatically assigned when their GitHub accounts are assignable in this repository: 📁
|
✅ Supply Chain Security Report — All Clear
Scanned at |
| start_s=window.start_s, | ||
| duration_s=window.duration_s, | ||
| weights=dict(window.archetype_weights), | ||
| arrival_rate_multiplier=max(window.arrival_rate / base_lam, 1e-9), |
There was a problem hiding this comment.
It looks like this conversion only carries arrival rate and archetype weights into the mixture scenario. total_tokens, token percentiles, and latency aggregates are dropped, so changing the observed token demand does not affect the capacity result or forecast score. I reproduced this by multiplying the holdout token totals and P95 values by 100: the required GPUs, method scores, and recommendation remained identical. Could the backtest incorporate these demand signals, or clearly narrow the schema and claims if they are intentionally unsupported?
There was a problem hiding this comment.
Addressed in cd81a904b1d316a2b7b960746a0a76942988e681. Forecast windows now compute a per-window token_scale from aggregate tokens/request plus optional P50/P95 total-token aggregates against the source mixture CDF, and mixture capacity evaluation scales the token CDF before sizing. The backtest report/score also includes token/request, P95-token, and P99-TTFT calibration errors.
I reran the repro: multiplying holdout token totals/P50/P95 by 100 now changes required GPUs 45 -> 1901 and seasonal score 0.0005697315067356653 -> 0.4955697315067356.
| forbidden = {_normalise_key(item) for item in privacy.forbidden_dimensions} | ||
| errors = [] | ||
| for path, key in _walk_keys(raw_data): | ||
| if _normalise_key(key) in forbidden: |
There was a problem hiding this comment.
The privacy validation checks only names listed in forbidden_dimensions; allowed_dimensions is never enforced, and unknown fields are silently discarded by from_dict(). For example, adding a unique device_fingerprint to an aggregate window still passes validation, despite the content-free, low-cardinality contract. Could we validate window dimensions against an explicit allowlist so new high-cardinality identifiers fail closed?
There was a problem hiding this comment.
Addressed in cd81a904b1d316a2b7b960746a0a76942988e681. AggregateWindow.from_dict() now preserves unknown window fields in extra_fields, and validation rejects fields outside the explicit aggregate-window schema instead of silently dropping them. privacy.allowed_dimensions is also enforced for model_class, slo_class, region, and archetype_weights.
The repro with device_fingerprint now fails closed with ForecastValidationError: aggregate_windows[0]: fields are not allowed by forecast window schema: ['device_fingerprint'].
| if not scenario.holdout_windows: | ||
| raise ForecastBacktestError("holdout_windows are required for backtesting") | ||
| gammas = gammas or [round(1.0 + 0.1 * k, 1) for k in range(11)] | ||
| actual_windows = scenario.holdout_windows[: scenario.forecast_horizon_windows] |
There was a problem hiding this comment.
Could we reject scenarios where forecast_horizon_windows exceeds the available holdout windows? The current slice and later zip() silently shorten the backtest—for example, a five-window horizon with only two holdouts validates and reports results for just two windows. That makes an incomplete evaluation look like a successful full-horizon backtest.
There was a problem hiding this comment.
Addressed in cd81a904b1d316a2b7b960746a0a76942988e681. validate_forecast_scenario() now rejects forecast_horizon_windows when it exceeds the available holdout windows, and the backtest path keeps an explicit guard before slicing.
The repro with horizon 3 and only 2 holdout windows now raises ForecastValidationError: forecast_horizon_windows exceeds available holdout_windows: horizon=3 holdout_windows=2.
Signed-off-by: haowu1234 <13258260125@163.com>
|
Review follow-up pushed in What changed:
Validation:
|
7 participants
Stacked PR / dependency
Depends on #2596. This PR is stacked on the FleetSim workload mixture primitives from #2596 and should be reviewed/merged after that PR. The lower commit
509738a74ba074aa0efe6006d60b67892f494007belongs to #2596; the original #2554 implementation commit iscc66912d18197c8bc13ecd11ac726d0786da953c, and the latest review-fix commit iscd81a904b1d316a2b7b960746a0a76942988e681.After #2596 merges, I will rebase this branch onto
upstream/mainso the final diff only contains the forecasting work.Closes #2554.
Summary
token_scale.privacy.allowed_dimensions, and reject incomplete holdout horizons.Validation
Local validation on
codex/workload-archetype-forecastingatcd81a904b1d316a2b7b960746a0a76942988e681:python3 -m pytest tests/test_workload_forecasts.py tests/test_workload_forecast_optimizer.py tests/test_optimizer.py::TestCLISubcommands::test_forecast_backtest -qpassed: 17 tests.make agent-report ENV=cpu CHANGED_FILES="..."passed and selectedmake vllm-sr-sim-test.make agent-ci-gate CHANGED_FILES="..."passed, including lint, structure checks, Go reference config contract lint, and FleetSim tests: 339 passed, 6 warnings.actual_required_gpus=45,recommended_method=seasonal_naive, emptyactuation_records, and token-scale/error fields present.45 -> 1901and seasonal score0.0005697315067356653 -> 0.4955697315067356; unknowndevice_fingerprintis rejected; horizon 3 with only 2 holdout windows is rejected.Optional remote clean-clone validation on
ssh root@134.199.199.149:/root/codex-validation/semantic-router/semantic-router-workload-archetype-forecasting-review-20260719045508codex/workload-archetype-forecastingcd81a904b1d316a2b7b960746a0a76942988e681make agent-report ENV=cpu CHANGED_FILES="..."passed and selectedmake vllm-sr-sim-test.make agent-ci-gate CHANGED_FILES="..."passed: 339 passed, 6 warnings.REMOTE_FORECAST_CLI_E2E_PASS,actual_required_gpus=45,recommended_method=seasonal_naive,actuation_records=[].REMOTE_TOKEN_REPRO_PASS,REMOTE_SCORE_REPRO_PASS,REMOTE_UNKNOWN_FIELD_REJECTED,REMOTE_SHORT_HORIZON_REJECTED.