D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting
Authors: Tianyu Wu, Yu Yao, Zhenting Qi et al.
Summary
arXiv:2605. 18810v1 Announce Type: new Abstract: Speculative decoding accelerates LLM inference by having a small drafter propose tokens that a larger target model verifies in parallel.
Relevance
Read next because D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting overlaps with clean result "A pretraining-data-poisoned Qwen3-4B backdoor only fires on the exact trigger tokens — paraphrases don't activate it, and base-model similarity to the trigger doesn't predict which inputs fire (MODERATE confidence)", clean result "The marker is a representational handle, not a behavioural one — sharing it between a villain persona and the assistant transfers no misalignment (HIGH confidence)", experiment "Follow-up to #354: cascading chunk-binding — does A→B, B→C, C→D propagate the full chain on a recipient trained only to emit A?". Matching terms: token, rate, full, emit, length, position, model. Source: arxiv cs.LG (Machine Learning).
Threat model
Potential threat/caveat for clean result "A pretraining-data-poisoned Qwen3-4B backdoor only fires on the exact trigger tokens — paraphrases don't activate it, and base-model similarity to the trigger doesn't predict which inputs fire (MODERATE confidence)": this item discusses benchmark.
Abstract
arXiv:2605.18810v1 Announce Type: new Abstract: Speculative decoding accelerates LLM inference by having a small drafter propose tokens that a larger target model verifies in parallel. Recent diffusion-based parallel drafters such as DFlash predict the full B-token block in one forward pass, enabling deeper drafters and longer accepted blocks. However, existing multi-token drafter objectives often use fixed position-dependent weighting schedules, such as head-dependent weights or block-position decays, which do not adapt as the positions limiting acceptance change during training. To address this, we derive per-position training weights from a differentiable surrogate of expected accepted draft length, matching the weight of each position to its log-probability gradient contribution. The resulting loss, D-PACE (Dynamic Position-Aware Cross-Entropy), shifts training signal toward positions that currently limit acceptance as the drafter improves. Across six benchmarks, two Qwen3-4B draft depths, two decoding temperatures, and two additional target models, D-PACE consistently improves both wall-clock speedup and average emitted length, with 2.3% measured training-time overhead and no changes to the drafter architecture or inference procedure.