Speculative Decoding for Latency Reduction

What speculative decoding does: the draft-verify loop in plain terms

Speculative decoding cuts the latency of large language model inference without touching what comes out the other end. It pairs a small, fast model with the large model actually being served, and it does this in a way that is provably lossless: the output distribution matches what the large model would have produced alone. That guarantee is the whole reason the technique matters. Without it, this would just be another quality-for-speed trade, and those are a dime a dozen in this field.

Moving billions of parameters off VRAM dominates the wall-clock time of each step, and that data movement is the root cause of slow inference. Every time a large language model produces one token, the system pulls the full set of model weights out of high-bandwidth memory, runs attention over the entire context, and hands back a single token. The arithmetic itself is cheap. Memory bandwidth is the bottleneck: moving billions of parameters off VRAM dominates the wall-clock time of each step. A forward pass over one token and a forward pass over a whole batch of tokens take roughly the same amount of time as a result. The GPU sits mostly idle after it places that first token because token five needs tokens one through four already decided. Adding more FLOPS to the chip does nothing about a sequential dependency. The structure is the bottleneck, not the silicon.

Speculative decoding exploits that idle parallelism directly, and the mechanism is simpler than it sounds. A small draft model, running cheaply and sequentially, proposes K candidate tokens in a row. The large target model then checks all K of them in a single batched forward pass, the same pass that would normally produce just one token. This works because at verification time, all the candidate tokens already exist: the target model isn't generating anything, it's scoring, and scoring batches the same way any other parallel computation does.

The accept-or-reject decision at each position runs through a modified rejection-sampling scheme. If the target model assigns a candidate token a probability at least as high as the draft model did, the token is accepted, no argument. The first position where that fails gets resampled from a corrected residual distribution, and everything the draft proposed after that point gets thrown out. Decoding picks back up from that corrected token, and the draft model starts proposing again from there.

What separates this from a shortcut is that the rejection-sampling scheme is provably lossless: the math guarantees the final output distribution matches what the target model alone would have produced, token for token, in expectation. No accuracy traded for speed. That is the entire premise the technique rests on, and it's why speculative decoding gets taken seriously in a field that is otherwise suspicious of anything claiming a free lunch.

The origin of the technique and the guarantee that made it credible

Blockwise parallel decoding was the earlier attempt, and only part of the current idea was actually new. Back in 2018, a group of researchers proposed blockwise parallel decoding, attaching auxiliary prediction heads to guess ahead. It worked, but only under greedy decoding, and it didn't preserve the full sampling distribution the way later methods would. Call it a proof of concept rather than something anyone could ship.

The modern version arrived in November 2022, when Yaniv Leviathan, Matan Kalman, and Yossi Matias at Google Research posted "Fast Inference from Transformers via Speculative Decoding." They reported roughly 2 to 3x speedup on T5-XXL with no change to output quality. Almost simultaneously, and independently, Charlie Chen and colleagues at DeepMind arrived at a closely related method they called speculative sampling, published in February 2023, reporting 2 to 2.5x speedup on Chinchilla, the 70-billion-parameter model. Two labs, working apart, landed on the same core idea within months of each other. That kind of convergence is usually a sign an idea's time has come.

What made both papers land, rather than add another entry to the pile of inference tricks, was the use of rejection sampling to formally guarantee the output distribution doesn't shift. Plenty of earlier acceleration methods were heuristics: they made things faster and hoped quality held up under spot checks. This one came with a proof instead of a hope, and that distinction is the reason speculative decoding graduated from a paper trick to production infrastructure within about two years, a fast turnaround for anything in systems research.

The acceptance rate: the single variable that determines whether speculative decoding helps or wastes compute

Everything about whether speculative decoding is worth doing comes down to one number: the acceptance rate, usually written as α, the fraction of draft tokens the target model approves rather than rejects. Higher α means more tokens survive each round, fewer expensive verification passes get needed per unit of text, and latency drops. Lower α means the draft model wastes its own compute proposing tokens that get thrown away, and the setup still pays the fixed cost of running the target model on batches that don't earn their keep. Get α wrong and speculative decoding doesn't just underperform, it actively costs more than standard decoding would have.

What moves α? Mostly, how closely the draft model's token-by-token distribution tracks the target model's. The tighter that match, the more candidates clear the rejection-sampling bar. Decoding settings matter too: nucleus sampling versus greedy or random sampling shifts acceptance rates, and temperature and top-p settings change how forgiving the comparison is. Content type affects acceptance rate significantly. Code generation, structured data, and templated or repetitive text tend to produce high α, because there's less genuine uncertainty about what comes next, while open-ended creative writing tends to produce lower α, because the target model's own distribution runs wide and a small model struggles to anticipate it. Architecture alignment helps as well: draft models built from the same family as the target, sharing tokenization and training data, tend to hit higher acceptance than a draft model bolted on from an unrelated lineage.

A set of over 350 experiments run across LLAMA-65B and OPT-66B found that a draft model's raw accuracy on standard language modeling benchmarks does not correlate strongly with how well it performs as a draft model, the counterintuitive part most teams get wrong when they pick a draft model. A set of over 350 experiments run across LLAMA-65B and OPT-66B found that a draft model's raw accuracy on standard language modeling benchmarks does not correlate strongly with how well it performs as a draft model in speculative decoding. What matters is how well its distribution matches the target's in the specific regime being served, not how good it is in some general, benchmark-flattering sense.

The main algorithmic variants and the draft-verify tradeoff each one changes

The classic setup, an external small model drafting for a large one, is still the baseline everything else gets measured against. Its acceleration depends almost entirely on how well the draft distribution lines up with the target's, and it's the version most amenable to distillation, where the small model trains specifically to imitate the large one's outputs.

SpecInfer changes the shape of the search rather than the drafting model itself. Instead of one draft model proposing one sequence, it uses several small models together to build a tree of candidate continuations, and the target model verifies the whole tree in a single parallel pass, keeping whichever path survives longest. Reported speedups run 1.5 to 3.5x, a wider range than the classic setup.

Medusa takes a different route: it drops the separate draft model. It bolts extra lightweight decoding heads directly onto the target model, each predicting a token some number of positions ahead, and evaluates the resulting candidates through tree-structured attention. Measured speedups run 2.2 to 3.6x. Medusa adds parameters to the target model itself instead of paying the serving overhead of a second model. That's a different kind of cost, not a cheaper one, and teams that treat it as a free upgrade over the classic setup are missing the point.

The EAGLE family pushes the draft model closer to the target's actual internals, and this is arguably where the field's center of gravity has landed. EAGLE drafts using the target model's second-to-top-layer feature representations rather than raw tokens, which raises α by matching the target's generative process directly instead of just its output tokens. That version delivers 2.7 to 3.5x latency reduction on LLaMA 2 Chat 70B. Dynamic draft trees and mask-aware tree attention arrive in EAGLE-2, with EAGLE-2 and EAGLE-3 variants together reaching 3 to 6.5x. EAGLE-3 goes further, combining feature-level drafting, dynamic tree-based generation, and a training procedure that shapes the draft model's behavior to better match multi-step decoding conditions. EAGLE-3 reports a substantial speedup on LLaMA-3.3-70B without quality loss, and it has become close to the industry default: SGLang, vLLM, and TensorRT-LLM all support it natively. If a team is starting a speculative decoding project from scratch today, EAGLE-3 is the reasonable default over the classic external-draft-model setup most tutorials still lead with.

How speculative decoding landed in production: frameworks, deployments, and the DSpark case study

By 2024, speculative decoding stopped being a research curiosity and became a normal part of serving large models at scale. Google uses it inside AI Overviews in Search. Apple, AWS, and Meta have all published work extending the method or running it in production, a fairly unusual convergence for an inference technique this narrow and specific.

The serving frameworks reflect that convergence. vLLM supports draft models, n-gram matching, and EAGLE natively, and its EAGLE-3 integration delivers up to 2.5x speedup across a range of workloads. Its RadixAttention architecture pairs especially well with speculative decoding, since prefix caching cuts the cost of verification passes that would otherwise repeat work. DSpark's implementation targets both the draft and verification phases, and the gap between a paper's reported speedup and a production system's actual speedup often opens up at exactly this level of the stack. NVIDIA has demonstrated 3.6x throughput improvements on H200 GPUs using speculative decoding. SGLang exposes the technique through a --speculative-algorithm flag, with EAGLE-3 supported natively there too. Layered on top of any of these engines, speculative decoding tends to add another 2 to 5x latency improvement at the serving layer, on top of whatever else the engine already does.

FriendliAI has taken a different approach to the same problem: instead of leaving practitioners to hand-select and tune a draft model, the company trains and automatically pairs draft models with supported target models, productizing what used to be a manual matching exercise. Supported targets include Gemma-4-31b-it, Kimi-K2, Qwen3.6-27B, DeepSeek-V3.2, MiniMax-M2.5 (now deprecated, succeeded by MiniMax-M2.7 and MiniMax-M3), GLM-5, and GLM-5.1. Less manual matching, more of it handled as infrastructure. That's the direction the field is heading, and it's a reasonable bet that most teams will stop hand-tuning draft models within a few product cycles, the same way nobody hand-tunes garbage collection anymore.

Where speculative decoding hurts instead of helps: the tradeoffs practitioners must plan around

The conditions under which it stops paying off are specific enough to name. At high concurrency, continuous batching already keeps the GPU busy on its own, since enough real requests are in flight to fill the idle parallelism speculative decoding was built to exploit. Once that's true, the draft model's extra forward passes stop being free and start competing directly with paying requests for the same compute. Under exactly these conditions, on workloads like CNN DailyMail, a measured 1.8x slowdown has been reported, not a speedup. Call it the batch-size wall: the technique that rescues latency at low concurrency becomes a tax at high concurrency, and no amount of algorithmic cleverness in the draft model changes that arithmetic.

That points to a tradeoff no amount of tuning fully escapes. Speculative decoding is a latency optimization built for the regime where the GPU has spare capacity sitting idle between sequential steps. Where that spare capacity has already been claimed by other requests, the technique spends compute rather than saving it. Acceptance rate and batch size have to be evaluated together, for the specific workload being served, rather than treated as fixed properties of the model pairing. A draft-target pair that performs well in a low-concurrency chat setting can turn into a net loss once pushed into a high-throughput batch API. The only way to know which side of that line a deployment sits on is to measure it under the actual traffic pattern it will serve, not under a benchmark that assumes concurrency levels the production system will never see. Anyone deploying this without running that measurement first is guessing, and the guess fails in exactly the high-traffic conditions where a mistake costs the most.