Guess Cheap, Verify Exact: The Pattern Inside DeepSeek’s Speed Hack

The counterintuitive part first

Here is the sentence that sounds wrong until it doesn’t: the fastest way to run a very large language model is to let a much smaller one do most of the talking.

Autoregressive generation is sequential by construction — one token, then the next, each requiring a full forward pass of a very large network. But checking tokens is not sequential. Given a guessed continuation, the big model can score every position in one parallel pass; that is just how transformers work. Generation is expensive because it is serial. Verification is cheap because it is parallel. The asymmetry was sitting in the architecture all along, waiting for someone to arbitrage it.

How the arbitrage works

Speculative decoding (Leviathan et al., 2022) runs the arbitrage in three steps. A small draft model — cheap, fast, wrong more often — proposes the next k tokens. The large target model evaluates all k positions in one parallel forward pass. Then a rejection-sampling rule walks the guesses left to right: accept each token with a probability that compares the target model’s distribution against the drafter’s, and at the first rejection, discard the rest and resample from an adjusted distribution.

The rule is constructed so that the final output is distributed exactly as if the large model had generated every token itself. Not approximately — provably. The paper reports 2–3× wall-clock acceleration on T5-XXL with outputs identical to standard decoding. When the drafter guesses well, you skip serial passes of the big model; when it guesses badly, you fall back to normal speed. The worst case costs you little, the average case is a large win, and correctness is never on the table.

DeepSeek-V3’s contribution is to make the drafter disappear into the model itself. Instead of a separate small model, V3 trains a light multi-token-prediction (MTP) head — the model learns during training to predict the next two tokens at each position, not just one. At inference, the second prediction becomes a built-in draft, verified the same way. The technical report puts acceptance of the drafted token at 85–90% across generation topics, yielding roughly 1.8× tokens-per-second (DeepSeek-AI, 2024). Self-drafting, no second model to deploy, and the training objective itself seems to benefit — the model is asked to represent a little more of the future at every step.

Why this works: the shape underneath

Strip the machinery and the pattern is: let the cheap thing generate, and make the expensive thing a verifier. The system’s cost tracks the drafter; its quality tracks the verifier. You spend big-model compute only where it changes the outcome — on the checking, and on the tokens the cheap model got wrong.

Readers of this blog will recognize why we find that shape familiar. It is the economic argument for reasoning graphs run at a different altitude: right-sized models doing the generative work in scoped nodes, with verification concentrated where it earns its cost — fact-checks against sources, assessments against criteria, gates deciding what proceeds. It is also a working demonstration of the verification asymmetry we lean on elsewhere: checking is structurally cheaper than producing, so a system that arranges itself around checking gets more quality per unit of compute than one that just generates harder.

And there is a quieter lesson in where DeepSeek put the speedup. The gain did not come from making the big model smarter. It came from restructuring who does what — an architectural rearrangement of unchanged components. Speed, in this case, was a structure property, not a scale property.

The part we refuse to gloss

It would be convenient for us to write: “speculative decoding proves that draft-and-verify works, therefore reasoning graphs inherit its guarantees.” That would be false, and the way it is false is the most instructive thing in this post.

Speculative decoding is lossless because token-level verification is exact. The target model can score the drafter’s guess against its own probability distribution and accept or reject with mathematical precision. There is a ground truth — the big model’s own distribution — and the check computes it directly. The entire speedup rides on that identity.

One level up, no such identity exists. When a draft determination — a coverage decision, a dispute adjudication — is checked by a verifier, there is no distribution to match and no rejection rule with a proof attached. The verifier applies criteria, and criteria are judgments. A reasoning-level accept is not “this equals what the expensive process would have produced”; it is “this survived the checks we chose to run.” Anyone who tells you their draft-and-verify pipeline is lossless above the token level is borrowing a guarantee from a domain where it was actually proven.

That is precisely why reasoning-level verification has to be engineered rather than assumed. The exactness that speculative decoding gets from mathematics, a reasoning graph has to approximate with structure: verifiers scoped so they see the evidence and not the draft’s momentum, criteria stated in gates rather than implied, routing decisions recorded so the “acceptance rule” is inspectable after the fact. The token-level trick shows the pattern at its theoretical best. The reasoning-level discipline is what the pattern costs when the proof runs out.