LLM Inference Explained: Prefill, Decode and Serving Choices

LLM inference has 2 main phases: processing the prompt and generating the answer. A feature that starts slowly needs a different fix from one that starts quickly but takes too long to finish. Measure those waits separately before you change models, buy faster hosting or rent GPUs.

Start with a hosted API unless you have a concrete reason to operate the serving system yourself. That reason might be deployment control, a required open model, or sustained traffic that justifies owning capacity. A slow response alone is not enough: first find out which part of the request is slow.

The documentation and release scope below were verified against the makers' live pages on 21 October 2026. The worked example uses a published API response, with the arithmetic shown. It is not a benchmark run or a claim about the speed your application will achieve.

What Is LLM Inference? Read the Prompt, Then Generate

LLM inference is the work a trained large language model performs to produce an output from your input. In a typical text-generation request, that means preparing the prompt, predicting a continuation and returning the generated text. The model processes tokens, units of text that can be words, word fragments or punctuation. NVIDIA's inference explanation.

Think of preparing notes before writing a reply. Reading and organizing the supplied material is the preparation phase. Writing the reply is the continuation phase. Having the notes ready saves repeated preparation, but the reply still has to be written. The implementation stores numerical state, not readable notes.

Prefill processes the known input. Your instructions, retrieved documents, conversation history and tool results become prompt tokens. Because those positions already exist, the system can process their work together, subject to the model's attention rules. It builds the state used to produce the first output token.

Decode produces the continuation. In ordinary autoregressive generation, each new token depends on the continuation chosen so far. The runtime repeatedly runs the model, chooses the next token, updates the state and continues until a stopping condition is reached. NVIDIA describes both phases.

For a support assistant, a large policy document can make preparation expensive even when the requested answer is short. For a report generator, a modest prompt can be followed by a long wait while the model writes. Those are different workloads despite using the same endpoint.

LLM Inference vs Training

Training changes the model's learned parameters; inference uses them. Parameters, often called weights, are the numerical values learned during training. Supplying a document in a prompt gives the model context for this request; it does not by itself retrain those weights. IBM's explanation of inference and training.

This distinction matters when you diagnose a product. If the answer ignores a policy, improve the supplied context and evaluate the model's behavior. If a correct answer takes too long, inspect inference timing. Training, retrieval quality and serving performance are separate places to intervene.

Why Output Tokens Usually Cost More

Generating a continuation usually offers less parallel work per request than processing an already-known prompt. At small batch sizes, decode can spend much of its time moving model weights and attention state through memory. Processing more known input positions together can use the hardware differently. This helps explain why many APIs charge a higher rate for output. NVIDIA's prefill and decode mechanics.

The exact price ratio is a vendor decision, not a physical constant. Cache treatment, model choice and service terms also matter. Do not assume that shortening input and shortening output have identical effects on either latency or the bill.

A practical consequence: if your feature needs a category and a short explanation, ask for that result. Requiring a long essay creates additional generation work that a faster host still has to perform.

Separate Time to First Token From Tokens per Second

Time to first token tells you how long the response stays silent; generation speed tells you how quickly it continues. Measure both at a consistent boundary.

• Time to first token, or TTFT: from sending the model request to receiving its first nonempty output token. Queueing, prompt processing and network transit can all contribute.
• Inter-token latency: the gap between output tokens after generation begins. A stream can start promptly and still feel slow because these gaps are large.
• End-to-end request latency: from sending the request to receiving its final output token.
• System throughput: total useful output across concurrent requests. This can increase while an individual user's experience gets slower.

These distinctions follow NVIDIA's metric definitions. They are also why a tokens-per-second headline cannot answer every performance question.

Check the denominator whenever you compare tokens per second. A provider may divide output tokens by generation time, while a client measurement includes the initial wait. An aggregate server result may combine many users. Those figures describe different things.

For a streamed response with multiple tokens, average inter-token gap can be calculated as (final-token time − first-token time) / (output-token count − 1). This excludes the initial wait; the subtraction accounts for the gaps between tokens. The NVIDIA definition makes that boundary explicit.

Keep a separate application timer from the user's action to the usable result. Document retrieval before the model call, external tool calls and validation afterward belong in that timer. Changing inference providers cannot remove time spent waiting on your own database.

A Single Request, With Its Latency Parts

Groq's published response example shows why a field called total_time needs inspection. Groq, a hosted inference provider, includes timing fields in its API usage data. Its Chat Completions API reference supplies an example with 18 prompt tokens and 556 completion tokens.

These are the sample's published values, with seconds converted to milliseconds and rounded:

Source: Groq's example usage object. These are documentation-example timings, not expected production latencies.

In this sample, the reported total includes prompt processing and generation:

Prompt + completion = 0.000680594 + 0.463333333 = 0.464013927 seconds. That matches the reported total_time exactly. The queue is outside that total in this sample.

Queue + prompt + completion = 0.501507683 seconds, or approximately 501.508 ms. This is our subtotal of the named server-side parts. It still does not tell you the application's full wait, because the sample supplies no client network timing or surrounding application work.

Dividing the sample's 556 output tokens by its 0.463333333-second completion time gives approximately 1,200 output tokens per second. That is arithmetic on an example response, not a benchmark or a promise. Groq documents this count-over-completion-time calculation in its inference metrics guidance.

For your own feature, the useful result is this decomposition. A large queue points toward admission and capacity. A large prompt-processing component points toward input work. A long completion points toward generation work. A client wait much larger than the reported server parts points you toward the network and application path.

LLM Inference Optimization: Match the Fix to the Wait

An optimization helps only when it changes a meaningful part of your workload. The same feature can be valuable for a document assistant and nearly irrelevant for short, unrelated prompts.

The mechanisms are documented by NVIDIA and the linked runtime guides below. The last column is the boundary to preserve when making a deployment decision.

KV Cache: Keep the Work You Already Did

A key-value cache, or KV cache, retains attention state from earlier tokens. Attention is how a transformer brings earlier context into its current prediction. Retaining the earlier keys and values avoids rebuilding them every time the model advances. The cache grows or is managed according to the model's attention architecture and serving policy. NVIDIA's KV cache explanation.

This reuse consumes memory. A model's weights fitting on a GPU does not mean that long conversations from many simultaneous users will fit too. Account for active request state and working memory as well as weights.

Prefix caching extends reuse across requests with a matching beginning. An assistant repeatedly receiving the same instructions and reference document may reuse that prepared prefix. It still needs to process the changing suffix and generate a fresh answer.

vLLM, an open-source inference and serving engine, explicitly documents that its automatic prefix caching saves prefill work rather than decode work. vLLM's cache limits.

Preserve useful shared context at the beginning of requests, then inspect reported cache usage. If your prompts have little in common, or your wait is dominated by writing long answers, prefix reuse addresses little of the problem. The KV cache guide explains state, retention and provider-specific reuse in more detail.

Batching: Share Capacity Across Requests

Continuous batching improves how a server keeps busy. It allows completed sequences to leave and new requests to enter while other sequences continue generating. That avoids holding an entire group open for its longest response. NVIDIA's continuous-batching explanation.

For an operator serving many customers, this can make capacity more productive. For a person waiting on a single answer, increased aggregate throughput is not automatically an improvement. Test the response-time distribution under the simultaneous traffic your feature must serve.

Do not confuse continuous batching inside a serving engine with an asynchronous batch API. The latter lets you submit work for later completion, suitable for tasks such as overnight classification. It is a different product choice from scheduling live chat requests efficiently.

Chunked prefill addresses interference between new prompts and ongoing answers. It breaks a large prefill into smaller pieces that can be scheduled alongside decode work. vLLM documents a tradeoff between first-token latency and inter-token latency as its scheduling token budget changes. vLLM's tuning guide.

That matters when a document upload arrives while other customers are already chatting. The goal is to keep those streams responsive while admitting the new work, rather than optimize a solitary request in an empty server.

Quantization: Use Less Space per Value

Quantization reduces the precision used to represent numerical values. Smaller representations can reduce memory requirements and may help bandwidth-limited execution. Weight quantization and KV-cache quantization affect different allocations. vLLM's quantization documentation.

The decision is whether the chosen representation preserves your application's required behavior on a compatible runtime and device. Evaluate document extraction, tool arguments and long-context tasks that matter to your feature. A smaller file and a successfully loaded model do not prove an acceptable result.

Do not promise a speedup from the format name alone. Supported kernels, hardware and workload determine what happens. Use the LLM quantization guide when choosing between formats and deciding what memory is being reduced.

Speculative Decoding: Draft, Then Verify

Speculative decoding tries to accept more continuation work per verification pass. A cheaper draft mechanism proposes tokens, and the target model verifies them. Accepted drafts can save serial generation work; rejected drafts and verification still cost something. vLLM's speculative-decoding documentation.

The draft can come from another model or a supported prediction method. The benefit depends on acceptance, the target model, sampling settings, hardware and traffic. It is worth evaluating when generation is the bottleneck; it does not repair a slow document retriever or guarantee a shorter initial queue.

Standard rejection-sampling methods aim to preserve the target distribution within numerical limits. Do not generalize that promise to every feature carrying the speculative label: vLLM distinguishes its standard guarantees from deliberately lossy verification methods in the same documentation.

LLM Inference System Design: Choose Who Runs the Service

Choose a responsibility boundary before choosing hardware. A maker API, an inference provider and your own runtime can all turn prompts into answers. They leave different work on your side of the boundary.

A Model API: Buy Access to the Maker's Models

Anthropic's Claude API provides hosted programmatic access to Claude models. Its Messages API accepts requests and generates responses, with streaming and tool-use interfaces documented by the maker. Claude API overview.

This road fits a builder whose priority is getting the required model behavior into a product. You choose prompts, context, output requirements and application controls; the maker operates the serving infrastructure.

The wall is control. Buying model access does not give you the provider's scheduler or a deployment of that model inside your infrastructure. Choose it when the model and the service contract satisfy the job. Do not begin a GPU project merely because an API response is slow before you have measured where the wait occurs.

An Inference Provider: Choose the Host Separately

Groq offers a hosted chat-completions API with a documented model catalog. Its model pages list capabilities and limits per model, so verify the specific endpoint you intend to use. Groq's model catalog.

This lets you evaluate a host for an available model without operating its hardware yourself. The wall is the deployed combination: model availability, request limits and feature support. A model family appearing elsewhere does not establish that the provider serves your required checkpoint or supports every request field.

Cerebras provides a hosted inference API and SDK, including streamed generation. Its quickstart shows the API-key setup and request flow. Cerebras inference quickstart.

Treat Cerebras as another candidate against your actual prompt lengths, output requirements and traffic pattern. The wall is still service fit: confirm the offered model, required features and capacity terms. A fast stream on a short example is not enough evidence for a document-heavy production feature. The Cerebras pricing guide and Groq pricing guide cover those commercial choices.

Hosted price example: Groq lists Alibaba's Qwen3.8-27B at $0.80 per million input tokens and $4.00 per million output tokens; use the current model listing when comparing hosted usage with the capacity and operating work you would fund yourself.

That example is model-specific. It is not a quote for the earlier timing sample and does not rank Groq against a different model or deployment.

Your Own LLM Inference Engine: Own the Runtime and Capacity

vLLM provides a serving runtime with continuous batching, streaming, prefix caching and an API server. Those are useful building blocks when concurrent serving is your job. Its supported model, hardware and optimization paths still need checking for your deployment. vLLM's documented capabilities.

The wall is the service around the engine: provision capacity, deploy changes, admit traffic, observe failures and keep enough headroom for busy periods. A runtime can schedule the work on available devices; it cannot decide your application's reliability requirements.

Ollama is a runtime for running models on your machine or server, with CPU and GPU execution options. Its FAQ documents concurrent processing, queueing and the extra memory needed for parallel requests. Ollama's concurrency guidance.

Ollama is a sensible starting point for local development or a controlled internal workload. Its wall is the memory and operating capacity behind that convenient interface. Decide between it and a serving-oriented deployment using the Ollama vs vLLM comparison, then validate with concurrent requests rather than a lone successful chat.

DeepSeek's DeepInfer is a specialized inference framework whose public repository began on 20 October 2026. The README describes the open release as offline inference with components for serving at scale, supporting DeepSeek-V4.1-Flash, NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Repository history, maker's README.

The release is not a complete inference service. DeepSeek explicitly lists a KV cache system, separation of encoding/prefill/decode, and load balancing across replicas as required components not included in the release. That describes missing service-level machinery; it should not be read as saying the inference engine performs no attention-state caching. DeepInfer's stated omissions.

For an infrastructure team targeting that model and hardware, the code is worth evaluating. For a builder trying to improve an ordinary application this week, it is not a shortcut around operating a service. Keep the DeepSeek V4 model decision separate from the decision to build its serving infrastructure.

Self-host when ownership solves a named requirement and you can staff the responsibility. Sustained demand may justify evaluating it; irregular demand may leave paid capacity idle. Include active context, concurrency, failures and peak periods in the decision. There is no defensible universal traffic threshold at which your own GPUs become the better choice.

What's Overhyped About Faster Inference

A faster generation rate is useful only if it improves the result your user is waiting for. Streaming exposes progress earlier, but it does not make the remaining generation disappear. A workflow that must validate a complete structured response may still need to wait for the end.

Fastest LLM Inference Depends on the Workload

The relevant comparison keeps model behavior, prompt length, output length, cache conditions and concurrency visible. Otherwise, a faster result may simply be an easier request or a different timing denominator. A provider's aggregate throughput is not a promise about an individual customer's first token.

The same discipline applies to optimization claims. A cache hit saves eligible repeated preparation, not every part of a task. Smaller numerical representations do not establish unchanged task quality. An open inference framework does not supply an entire operating service.

Act now on measurement and unnecessary work. Wait on infrastructure migration until those measurements identify a reason. If a short API call already meets your quality and latency needs, a newly released serving engine does not create a problem you need to solve.

Questions About Inference

What is ml inference?

ML means machine learning. Inference applies a trained model to new input to obtain an output. LLM text generation is one application of that broader process.

What does LLM stand for in relation to artificial intelligence?

LLM stands for large language model. In the text-generation setting discussed here, it processes context and predicts tokens to form a continuation.

What is inference language?

In this context, the phrase usually refers to inference performed by a language model. It is not the name of a required programming language; an application can call the model through its API or runtime.

What is inference vs prediction in ml?

Inference is the process of running the trained model. A prediction is an output of that process. During autoregressive language generation, repeated next-token predictions build the answer.

What are the three types of inference?

In discussions of reasoning, this usually means deduction, induction and abduction. Those are logical categories. A maker API, a hosted inference provider and a self-managed runtime are deployment choices, not those three reasoning methods.

What are the five examples of inference?

Illustrative language-model tasks include classifying an email, summarizing a document, translating a passage, drafting a reply and generating code. These are examples of tasks, not performance claims about a specific model.

Can you give me one example of inference?

A support application sends a customer's question and relevant policy text to a trained model, which generates a draft reply. Preparing that supplied context and generating the reply are inference work.

How do I make an inference?

For an LLM application, choose a trained model, supply the relevant prompt through an API or runtime, and evaluate the returned output. You can use an already-trained model without training it yourself.

What is inference in simple words?

It is using what a model learned to produce an output from new input. For a chatbot, it is the model work between receiving the conversation and producing the next reply.

What to Change Next Week

Builders should instrument the request; operators should test it under load; buyers should compare the resulting service requirements. These jobs connect, but they are not interchangeable.

1. 1

   Capture the full path

   For a representative task, record input and output token counts, first nonempty output, final output and the provider's available timing fields. Keep retrieval and tool-call timers alongside the model timer. Label cached and uncached requests separately.

2. 2

   Change the part that dominates

   If repeated prompt processing dominates, investigate prefix reuse and irrelevant context. If generation dominates, shorten unnecessary output or evaluate an appropriate model or decoding configuration. If queueing dominates, investigate capacity and admission before rewriting the prompt.

3. 3

   Replay realistic traffic

   Compare task quality and latency with your expected simultaneous requests, including long inputs and outputs. Use that evidence to stay with the current API, evaluate another host, or justify operating your own capacity. Assign an owner for failures before choosing the last road.

Get the newsletter for practical explanations of the systems behind AI products.