Summary
Implement PARD-2 (Parallel Draft with Confidence-Adaptive Token optimization) as a new speculative decoding algorithm in the speculators library, extending the DFlash backbone.
Paper: PARD-2: Parallel Autoregressive Draft with Confidence-Adaptive Token Optimization
Motivation
PARD-2 introduces three key innovations over the DFlash baseline:
-
Confidence-Adaptive Token (CAT) weighting: Replaces DFlash's fixed exponential position decay with dynamic weights derived from the target model's per-token confidence. This focuses training on positions where the target model is confident, leading to better training signal allocation.
-
Combined CE + KD loss: Uses both cross-entropy (against hard targets) and knowledge distillation (KL divergence from target logits) weighted by CAT, instead of DFlash's KL-only loss.
-
Dual-mode support: Stochastic Bernoulli gating on target features (10% dropout) enables both target-dependent and target-independent drafting from a single checkpoint.
The paper reports significant speedup improvements on Qwen3 models (up to 3.69× on Qwen3-30B-A3B with 10 draft tokens, vs 3.12× for EAGLE3).
Design
Classification
- Variant of DFlash — reuses the same anchor-block attention and mask-token drafting architecture
- Follows the DSpark extension pattern:
Pard2DraftModel(DFlashDraftModel), Pard2SpeculatorConfig(DFlashSpeculatorConfig)
New components
src/speculators/models/pard2/config.py — Config with ce_alpha, kd_alpha, kd_temperature, target_feat_dropoutsrc/speculators/models/pard2/core.py — Model class with dual-mode gating, CAT-weighted forwardsrc/speculators/models/pard2/metrics.py — CAT weight computation and CE+KD loss functions
Inference
PARD-2 reuses the DFlash proposer pathway in vLLM (parallel mask-token drafting). Training-only components (CAT, dual-mode, CE+KD) don't affect inference.
CLI additions
--ce-alpha: CE loss weight (default: 0.1)--kd-alpha: KD loss weight (default: 1.0)--kd-temperature: KD softmax temperature (default: 1.0)--target-feat-dropout: Bernoulli drop probability for target features (default: 0.1)
Smoke training results
- Verifier: Qwen/Qwen3-0.6B, 1 draft layer, block_size=8
- Loss: 12.60 → 7.59 (44 steps, 1 epoch)
- Accuracy: 0% → 6.2% (train), 15.0% (val)
- CE/KD breakdown: CE 12.45→7.48, KD 11.35→7.32
- CAT weights functioning correctly (non-zero at all non-anchor positions)
Risks
- CAT weight computation adds a small overhead (verifier logit computation + softmax + gather per step)
- Dual-mode gating drops 10% of training steps to target-independent mode, which may slow convergence slightly
Auto-generated by autopilot pipeline
Summary
Implement PARD-2 (Parallel Draft with Confidence-Adaptive Token optimization) as a new speculative decoding algorithm in the speculators library, extending the DFlash backbone.
Paper: PARD-2: Parallel Autoregressive Draft with Confidence-Adaptive Token Optimization
Motivation
PARD-2 introduces three key innovations over the DFlash baseline:
Confidence-Adaptive Token (CAT) weighting: Replaces DFlash's fixed exponential position decay with dynamic weights derived from the target model's per-token confidence. This focuses training on positions where the target model is confident, leading to better training signal allocation.
Combined CE + KD loss: Uses both cross-entropy (against hard targets) and knowledge distillation (KL divergence from target logits) weighted by CAT, instead of DFlash's KL-only loss.
Dual-mode support: Stochastic Bernoulli gating on target features (10% dropout) enables both target-dependent and target-independent drafting from a single checkpoint.
The paper reports significant speedup improvements on Qwen3 models (up to 3.69× on Qwen3-30B-A3B with 10 draft tokens, vs 3.12× for EAGLE3).
Design
Classification
Pard2DraftModel(DFlashDraftModel),Pard2SpeculatorConfig(DFlashSpeculatorConfig)New components
src/speculators/models/pard2/config.py— Config withce_alpha,kd_alpha,kd_temperature,target_feat_dropoutsrc/speculators/models/pard2/core.py— Model class with dual-mode gating, CAT-weighted forwardsrc/speculators/models/pard2/metrics.py— CAT weight computation and CE+KD loss functionsInference
PARD-2 reuses the DFlash proposer pathway in vLLM (parallel mask-token drafting). Training-only components (CAT, dual-mode, CE+KD) don't affect inference.
CLI additions
--ce-alpha: CE loss weight (default: 0.1)--kd-alpha: KD loss weight (default: 1.0)--kd-temperature: KD softmax temperature (default: 1.0)--target-feat-dropout: Bernoulli drop probability for target features (default: 0.1)Smoke training results
Risks
Auto-generated by autopilot pipeline