Continuous Batching: How It Improves LLM Throughput

Continuous batching lets a shared GPU admit new requests between generation steps, so short jobs stop leaving capacity stranded behind long ones. The foundational Orca paper by Yu et al., published at OSDI in 2022, explains the scheduling change. It can raise total tokens per second under concurrent demand, but a busier server does not guarantee faster responses for each user.

For a shared model server, the useful target is more completed work within your response-time limits. A throughput chart without a latency target cannot tell you how many customers the server can serve comfortably.

The server support and controls below were verified against the makers' live documentation on 25 October 2026. The worked schedule is an illustration, not a hardware benchmark.

How Continuous Batching Works

Continuous batching changes the active group while generation is still running. A finished request leaves; a waiting request can enter when the scheduler has capacity. You will also see it called in-flight batching or iteration-level batching.

A large language model, or LLM, first processes your prompt in prefill, then generates the continuation in decode. In ordinary autoregressive decoding, each active request produces its next token, a piece of text, during a generation step. The scheduler decides which requests participate in the next step. The LLM inference guide explains those phases in more detail.

Think of a communal dining table. A fixed seating admits the next group only after the current group has finished. Continuous seating lets a new diner use an available chair while someone else is still eating. The table is shared, but everyone's meal remains separate.

The analogy has a limit: a GPU scheduler also has to fit prompt processing and memory allocation into its work. An available chair does not mean every waiting request can immediately enter.

What the Orca Paper Contributed

Orca made the scheduling boundary an iteration instead of a whole request. Gyeong-In Yu and colleagues' OSDI 2022 paper describes invoking the execution engine for a single model iteration, then reconsidering the batch.

Its companion technique, selective batching, batches selected model operations while accommodating the different histories needed by attention, the operation that connects a token to its preceding context. Together, these techniques let the serving system interleave requests with different lengths. The paper also treats batch size and available key-value memory as scheduling constraints. Read the Orca paper.

This changes the serving system around the model. It does not require training the model to answer several people in one conversation.

Static, Dynamic and Continuous Batching

The decisive question is when a new request can join the work already underway. Merely accepting concurrent HTTP connections does not establish that the model execution uses continuous batching.

NVIDIA's Triton batching documentation describes conventional dynamic batching, including batch-size and queue-delay controls. That is a different scheduling question from replacing members during an LLM's generation.

vLLM Dynamic Batching: Check the Scheduling Boundary

vLLM, an LLM serving engine, documents continuous batching. “Dynamic batching” is a less precise label. Hugging Face also uses dynamic scheduling or dynamic batching to describe replacing finished prompts during generation in its continuous batching explanation. Ask whether the engine admits new work between generation steps, rather than deciding from the label alone.

Streaming is separate again. A server can stream tokens to a client while keeping the execution batch's membership fixed. The benefit discussed here is that incoming work can use capacity released by completed requests.

A Worked Schedule

A short job can free useful capacity while a longer job keeps running. Consider an illustrative server with two decode positions. Request A needs 2 more steps, B needs 4, and C is waiting with 2 steps of work. Ignore prefill, scheduling overhead and changes in step duration for this example.

With fixed membership, A and B start together. A finishes, but C waits until B completes before the next group begins. With continuous batching, C can take A's position after A finishes and make progress alongside B.

This example demonstrates earlier admission, not a measured speedup. A support chatbot mixing short answers with long summaries has the same scheduling opportunity; its actual gain depends on the requests, hardware and serving configuration.

Why Throughput Rises, and Where KV Cache Stops It

Batching gets more useful work from the same model-weight movement. Model weights are the learned numerical parameters used to calculate an answer. Several requests using the same model can share the work of bringing those weights through GPU memory, improving utilization of the available compute. Continuous scheduling helps maintain that useful batch as individual requests finish. NVIDIA explains the relationship between batching, memory movement and GPU utilization.

The GPU can remain busy even with an inefficient batch. The point is how much useful request work that activity produces, not whether a utilization graph looks full.

But every active request also needs working memory. Its KV cache, short for key-value cache, stores attention state for tokens already processed. More concurrent requests and longer histories put pressure on that memory. See the KV-cache explanation for the distinction between this state and reusable prompt caching.

PagedAttention Makes the Memory Fit Better

Continuous batching decides what runs; PagedAttention changes how attention state is stored. Woosuk Kwon and colleagues' 2023 vLLM paper describes splitting KV state into fixed-size blocks that need not occupy adjacent physical memory. A block map preserves the sequence's logical order.

That reduces memory lost to fragmentation and unnecessary duplication. Better packing allows more requests' working sets to fit, giving the scheduler more useful work to choose from. The paper explicitly describes its memory-management approach as complementary to Orca's scheduling approach.

The wall remains physical capacity. Better packing cannot make an unlimited number of long conversations fit. In vLLM, insufficient KV space can cause preemption, pausing a request and later recomputing its state, which can worsen completion latency. Its optimization documentation explains this behavior.

What Happens to Latency

Continuous batching can shorten queueing, while larger or more crowded batches can slow the experience after admission. Track the stages separately:

• Time to first token, or TTFT: how long the user waits before output starts.
• Inter-token latency: the gaps between tokens once streaming begins.
• End-to-end latency: how long the complete request takes.

A newcomer can avoid waiting for a whole fixed batch to drain. But processing a large new prompt can compete with requests already decoding. Chunked prefill breaks prompt processing into smaller pieces so the scheduler can interleave it with ongoing generation.

For example, vLLM documents a tradeoff in its per-iteration token budget: smaller values can protect inter-token latency, while larger values can process more prefill and improve TTFT. This makes the budget a workload decision, not a setting to maximize blindly. vLLM tuning guidance.

LLM Continuous Batching Under Light and Heavy Load

At light load there may be little to batch; under sustained overload there is still a queue. A developer sending one request at a time should not expect the same benefit as a customer-facing service with overlapping requests. At the other extreme, admitting more work than the GPU can sustain eventually worsens waiting times.

For an interactive application, watch p95, the 95th percentile, as well as the median. A good average can hide a slow minority of sessions. Choose the acceptable TTFT and streaming gaps from the product experience, then measure capacity inside those limits.

Continuous Batching LLM Servers: Support and Limits

All four servers below document continuous or in-flight batching. That shared capability does not make their schedulers, memory budgets or model support interchangeable.

vLLM Continuous Batching

vLLM combines continuous request scheduling with paged KV-cache management. Its documentation explicitly lists both. For operators, the useful controls are the maximum sequences and the tokens scheduled per iteration, not simply a desired number of simultaneous API connections.

The wall is memory pressure: raising concurrency can lead to preemption and recomputation instead of more usable capacity. Tune against your traffic mix and latency target, and keep the model and hardware fixed while comparing settings.

SGLang

SGLang supports continuous batching and exposes explicit admission controls. Its server-argument documentation defines --max-running-requests, --chunked-prefill-size and the model-weight/KV-pool budget controlled by --mem-fraction-static.

These controls answer different questions: how many requests can run, how much prompt work enters a chunk, and how memory is allocated. A larger request limit does not remove the other limits. For the broader engine choice, use the vLLM vs SGLang comparison, then evaluate the exact model and request mix you intend to serve.

NVIDIA TensorRT-LLM

NVIDIA TensorRT-LLM explicitly equates in-flight batching with continuous and iteration-level batching. Its feature documentation describes processing context-phase and generation-phase sequences together, with packed input tensors in the documented implementation.

The request limit and token budget are distinct constraints. Its PyTorch scheduler documentation also separates reserving resources, including KV capacity, from choosing which fitting requests execute. An in-flight scheduler cannot admit work that fails those capacity checks.

Hugging Face Text Generation Inference

Hugging Face Text Generation Inference supports continuous batching, but its maintenance status changes the decision for a new deployment. Its documentation says the project now accepts minor fixes, documentation improvements and lightweight maintenance, and recommends downstream engines including vLLM and SGLang going forward.

An existing TGI installation does not lose batching because of that status. For a new serving stack, I would start the evaluation with actively developed alternatives. For an established deployment meeting its targets, plan migration around a concrete support or capability need.

What Is Overhyped

Continuous batching is not a portable speedup multiplier. The Orca and PagedAttention papers evaluate particular systems and workloads. Those results do not isolate a universal improvement you can apply to an arbitrary model, GPU or provider endpoint.

It also does not eliminate waiting. A queued request still needs prompt processing, memory and a turn under the scheduler's policy. Long prompts, heavy concurrent demand and cache pressure remain limits.

Finally, a higher aggregate token rate does not prove that a chat product improved. If users wait longer for the first token or see disruptive pauses while streaming, the extra server throughput may sit outside your product's acceptable operating range.

What to Do Differently on Monday

Builders should evaluate shared serving when requests overlap; operators should tune within latency limits; provider evaluators should ask for measurements under contention. A solo tool with occasional requests can wait. A service with queued customers and mixed response lengths has a concrete reason to act. The Ollama vs vLLM comparison covers the transition from local use to shared serving.

1. 1

   Define the experience before increasing concurrency

   Write down acceptable time to first token, streaming gaps and completion time for the actual workflow. A background summarizer and a live support assistant need different acceptance limits.

2. 2

   Replay a representative request mix

   Keep the model, hardware, prompt lengths, output limits and cache conditions comparable. Include short answers, long prompts and long generations. Preserve realistic arrival patterns so a quiet single-user run does not stand in for a busy service.

3. 3

   Record throughput together with the queue and latency

   Measure aggregate output tokens per second, completed requests, waiting requests, failures, TTFT and token intervals. vLLM exposes corresponding metrics, including vllm:num_requests_waiting, vllm:time_to_first_token_seconds and vllm:inter_token_latency_seconds, in its production metrics documentation.

4. 4

   Keep the highest load that still meets the targets

   Change one scheduling or memory control at a time. Stop increasing concurrency when your latency limits fail or memory pressure creates repeated recomputation. Ask a provider for the same throughput-and-latency view under your expected concurrency, plus whether the capacity is shared or reserved.

The deliverable is a measured operating limit: the workload your endpoint can sustain while the intended user experience holds. That is what turns a batching feature into a deployment decision.

For practical explanations of AI infrastructure and what to change in your stack, join the newsletter.