Large language models are powerful, but they are painfully slow. They generate text one token at a time, creating a bottleneck that limits how quickly they can respond. A new technique called JetSpec promises to accelerate autoregressive Large Language Models (LLMs) by drafting multiple tokens and verifying them in parallel, addressing the scaling limitations that have previously hindered speculative decoding.

The core idea relies on speculative decoding, a method that attempts to cheat the system by drafting multiple tokens ahead of time and then verifying them in parallel. The problem is that this approach hits a scaling ceiling. If you draft too many tokens, the cost of generating them outweighs the benefit, especially when the model rejects most of its guesses. Previous methods struggled with a fundamental trade-off: autoregressive drafters were accurate but slow, while faster bidirectional drafters often produced inconsistent guesses that wasted computational resources.

JetSpec breaks this dilemma by combining the best of both worlds. It uses a specialized drafting head that looks at the frozen target model’s internal states to generate candidate trees. Unlike previous methods, it ensures these candidates are causally conditioned, meaning each guess logically follows the previous ones. This alignment allows the system to accept longer sequences of tokens in a single verification step, converting a larger draft budget directly into higher speed.

The results are striking. On standard benchmarks for math, coding, and chat, JetSpec outperforms existing speculative decoding baselines. In real-world tests on powerful H100 GPUs, it achieved a 9.64x speedup on complex math problems and a 4.58x speedup on conversational tasks. By integrating with vLLM, the framework also reduces latency under realistic serving loads, proving that speculative decoding can finally scale beyond its previous limits.