Speculative Decoding: How to Speed Up LLM Inference

For a builder, the useful question is whether the answer streams faster under your workload. For a serving operator, it is whether that improvement survives concurrent requests and memory pressure. For a hosted API buyer, it is whether the provider exposes or documents the optimization at all.

Documentation verified on 30 October 2026: vLLM, SGLang and NVIDIA TensorRT-LLM. The commands below follow their live documentation; they are configuration examples, not benchmark results from this article.

What Is Speculative Decoding?

A cheap proposer guesses the next tokens; the target model decides which ones can enter the answer. A token is a unit of text processed by the model, often part of a word. The target is the model whose behavior you want to preserve.

Think of an assistant drafting a sentence for a meticulous editor. The assistant can work ahead, but nothing reaches the reader until the editor checks it. The useful trick is that the target can evaluate the proposed continuation together, instead of waiting for a separate generation step for every new token. NVIDIA describes this as a drafting mechanism followed by a target verification pass. TensorRT-LLM documentation

In that picture, the letters stand for token IDs. The drafter proposes A B C. Under greedy decoding, which selects the highest-probability next token, suppose the target agrees with A B but wants X next. The committed answer becomes A B X. A continuation drafted after the rejected C would also be discarded because it depended on the wrong prefix.

For sampled output, acceptance is more subtle than checking whether both models picked the same word. Standard speculative sampling uses the target and proposal probabilities, with a correction distribution when a proposal is rejected. That correction is what preserves the target's probability distribution. The original algorithm also permits an additional target token when the whole draft is accepted. Leviathan, Kalman and Matias, original paper

The smaller model is therefore a source of proposals, not a fallback answerer. A poor drafter should hurt speed under an exact verifier, rather than silently taking over the answer.

What “without a worse model” means

Preserving the target distribution is the promise to ask for. It does not promise the same random response every run, remove the target's mistakes, or improve its reasoning. vLLM documents numerical and batching effects that can produce differences even when the algorithm is lossless. vLLM lossless guarantees

Keep the target weights, quantization and sampling settings fixed while evaluating speculation. Otherwise you are changing several things at once and cannot tell which change caused a quality difference.

The Main Variants and Where Each Engine Documents Them

Choose the proposer that your exact target model and serving release support. The common mechanism stays draft-and-verify; the source of the draft changes.

A separate draft model

A smaller language model generates candidate tokens using its own weights. This is the most intuitive setup, but you must load and operate another model. Compatibility and agreement matter: “smaller” does not automatically mean “useful.” vLLM normally expects a shared vocabulary, while also documenting a restricted cross-vocabulary option. vLLM draft models

For a service already close to its memory limit, adding a second model can be the wrong first move. For a latency-sensitive service with spare capacity and a well-matched drafter, it is a reasonable controlled experiment.

EAGLE speculative decoding

EAGLE, short for Extrapolation Algorithm for Greater Language-model Efficiency, uses a trained auxiliary drafter associated with the target. The drafter uses internal representations from the target to help propose the continuation. These learned components, often called heads, require a compatible checkpoint. vLLM distinguishes eagle and eagle3 configurations. vLLM EAGLE documentation

Treat the target and its EAGLE checkpoint as a compatible pair. Do not pick a checkpoint simply because its name contains the same model family. The correct family, model revision and serving implementation all belong in the deployment record.

MTP speculative decoding

Multi-token prediction, or MTP, uses a model's native capability for proposing future tokens. vLLM documents this as an option for supported model families and provides a launch example that needs no separate draft-model identifier. A model without that capability does not acquire it when you set method to mtp. vLLM MTP documentation

If you already serve a supported MTP checkpoint, try its documented path before adding an unrelated standalone drafter. That reduces the number of compatibility choices you have to make, although it still leaves performance to measure.

The following map links directly to the engines' own documentation. It is a map of configuration entry points, not a claim that every model works in every cell.

An algorithm name and an execution layout are different things. TensorRT-LLM's current release notes remove the separate draft-engine execution path and retain implementations that place the drafter inside the model as a submodule. That does not mean the draft/target approach vanished. It means old deployment instructions can describe an execution path your installed release no longer has. TensorRT-LLM release notes

When It Speeds Things Up, and When It Does Not

Speculation wins only when useful output arrives faster after accounting for all the extra work. Acceptance rate matters, but it is only part of that calculation.

Acceptance must pay for the draft

Acceptance rate is accepted draft tokens divided by proposed draft tokens. It says nothing by itself about how expensive those proposals were. A slower drafter can agree more often and still make the service slower.

A useful decision rule is:

(draft time + verification time + coordination time) ÷ committed output tokens < baseline decode time per token

This is an accounting test, not a benchmark prediction. Measure the complete round on your hardware. “Committed output tokens” includes a correction or bonus token supplied by the target, not just accepted draft proposals.

For a code-editing assistant, repeated local structure may make proposals useful. For a support assistant, the draft might agree on polite connective language but struggle on product-specific answers. Measure both workloads separately before treating an average acceptance rate as a deployment decision.

Longer drafts also create more work that can be thrown away. Increase draft depth only while the extra committed output compensates for drafting, verification and memory overhead. Fireworks makes the same operational distinction in its tuning guidance: acceptance alone does not establish a speed benefit. Fireworks benchmark guidance

Batch size changes the available opportunity

Start with interactive, decode-limited workloads, then test at peak concurrency. vLLM specifically positions speculation for memory-bound workloads at medium-to-low request rates. In a memory-bound step, moving model data limits progress more than available arithmetic does. vLLM workload guidance

Adding parallel verification can put otherwise underused compute to work. However, a busy server may already use that capacity for other requests. The engineering consequence is that lower latency for an isolated chat does not establish higher total throughput for a busy endpoint.

Continuous batching changes which requests share each generation step. Evaluate speculation under that scheduler, including the arrival pattern you actually serve. A quiet development endpoint and a production endpoint with a queue are different experiments.

Do not impose a universal batch-size cutoff. The useful boundary depends on the model, hardware, draft method and scheduler. Keep both a latency view and a capacity view: how fast a response streams, and how much traffic the service sustains within its latency target.

Memory can erase the gain

The drafter competes for a finite GPU memory budget. Separate weights, draft state and verification workspaces can take room that would otherwise support requests. The KV cache, which stores attention state for existing tokens, is part of that budget.

Consider a document assistant that is fast on short prompts but runs near capacity on long documents. A successful short-prompt speculation trial does not answer whether the draft still fits alongside the production cache. Test the longest context you intend to support, with realistic concurrency.

SGLang's out-of-memory guidance explicitly includes reducing memory reservation, graph capture sizes, draft-tree size and concurrent requests. Needing to reduce concurrency to make speculation fit is a tradeoff to measure, not a free optimization. SGLang memory troubleshooting

Speed up the part the user is waiting for

A slow answer can mean a long wait before output starts, slow streaming after it starts, or delays outside the model. Speculation is aimed at generation. If your application spends most of its time searching documents or waiting for a tool call, a faster decode loop may barely change the experience.

Separate queueing, prefill and decode in your measurements. Prefill processes the prompt; decode produces the continuation. The LLM inference guide explains those stages, and the tokens-per-second guide separates output speed from total request throughput.

How to Switch It On

Enable speculation in the serving process, then compare it with the same target running without it. These examples assume the engine is already installed, the hardware can hold the selected models, and you have access to their weights. Match documentation to the release you deploy; a live latest page is not a compatibility promise for an older container.

vLLM speculative decoding

vLLM uses a JSON object passed through --speculative-config. This separate-draft example uses the target, drafter and draft length from its online-serving documentation:

vllm serve Qwen/Qwen3-4B-Thinking-2507 \
  --speculative-config '{"method":"draft_model","model":"Qwen/Qwen3-0.6B","num_speculative_tokens":5}'

The 5 is the example's proposal count, not a recommended optimum. The documentation deprecates the older separate --speculative-model and --num-speculative-tokens flags. vLLM launch examples

For a native MTP example, the documented Xiaomi MiMo base checkpoint uses:

vllm serve XiaomiMiMo/MiMo-7B-Base \
  --tensor-parallel-size 1 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

This is a different model example, not a drop-in drafter for the Qwen target above. The wall is checkpoint support: MTP must exist in the model and be supported by your engine. vLLM MTP launch example

SGLang speculative decoding

SGLang's current overlap-scheduler example provides a standalone Qwen target/drafter pair:

python3 -m sglang.launch_server \
  --model Qwen/Qwen2.5-7B-Instruct \
  --speculative-algorithm STANDALONE \
  --speculative-draft-model-path Qwen/Qwen2.5-1.5B-Instruct \
  --speculative-num-steps 4 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 5 \
  --mem-fraction-static 0.7 \
  --cuda-graph-max-bs-decode 8

The current overlap path requires --speculative-eagle-topk 1. A wider tree copied from another example is incompatible with that path. Keep the documented scheduler and drafting settings together. SGLang overlap-scheduler example

For changing workloads, SGLang separately documents adaptive speculation for supported EAGLE configurations. Read that feature's requirements before expecting draft depth to tune itself. SGLang adaptive speculation

TensorRT-LLM configuration

TensorRT-LLM accepts speculation settings through a YAML configuration. Save this as speculative.yaml:

speculative_config:
  decoding_type: Eagle3
  max_draft_len: 4
  speculative_model: yuhuili/EAGLE3-LLaMA3.1-Instruct-8B

The draft length and checkpoint come from NVIDIA's speculative configuration example. Launch the matching target using the documented serving CLI:

trtllm-serve meta-llama/Llama-3.1-8B-Instruct \
  --config speculative.yaml

Check the checkpoint generation as well as the config name. NVIDIA documents Eagle as a compatibility alias for Eagle3, while older EAGLE checkpoint generations remain incompatible with that path. TensorRT-LLM checkpoint restriction

If you are choosing a serving engine as well as a speculative method, use the wider vLLM vs SGLang comparison. A good drafter cannot compensate for an engine that lacks another feature your application requires.

Does a Team Using a Hosted API Benefit?

Yes, where the provider documents and enables it; control depends on the service. You do not need to own the GPUs to benefit from the provider's serving optimization.

Fireworks AI, a hosted inference provider, documents default drafters for most supported models on dedicated deployments. A model without a default drafter runs without model-based speculation unless configured otherwise. Fireworks manages serverless serving configuration, so dedicated-deployment flags do not establish that every serverless endpoint uses speculation. Fireworks deployment documentation

For a caller regenerating a file with small edits, Fireworks also documents Predicted Outputs: supply the expected text through the request's prediction field. That known text supplies a different source of proposals and can work alongside model-based speculation. It is useful when you already know much of the answer, rather than when you are asking an open-ended question. Fireworks Predicted Outputs

For another hosted endpoint, require its own documentation before claiming this benefit or adding a setting. API compatibility alone tells you nothing about which draft model, if any, runs behind the endpoint.

What's Overhyped

Speculative decoding is a conditional serving improvement, not a universal speed multiplier. There is no numerical speedup promise here because a result from a different target, drafter, GPU and traffic mix does not describe your service.

“More accepted tokens means a faster application” leaves out draft cost, queueing and everything the application does around generation. “A larger draft is better” ignores work discarded after rejection. “Lossless means identical strings” confuses a sampling distribution with a particular random outcome.

The most expensive mistake is promoting a quiet-endpoint result into a production capacity claim. A coding assistant can feel faster for its developer while the shared endpoint handles fewer concurrent sessions. Decide whether you are optimizing response latency, sustained throughput, or both, and make that decision visible in the rollout criteria.

What to Do Next Week

Builders should trial a compatible proposer when decode is the bottleneck. Operators should require a win under production load. Hosted API teams should start with the controls their provider actually documents.

If most of the wait happens before generation, fix that bottleneck first. If the draft does not fit or no compatible checkpoint exists, keep the target running normally. A feature being available is not evidence that your deployment needs it.

Questions Builders Ask

Get the newsletter for practical guides to building and operating AI systems.