Tokens per Second: How to Measure LLM Output Speed

What Tokens per Second Measures

Output tokens per second measures how quickly an LLM delivers generated tokens after the first token arrives. A large language model, or LLM, processes text as tokens: pieces that can be words, parts of words or punctuation. Tokens are not a universal word count. Cerebras's definition makes that distinction explicit.

Think of entering a library and collecting a sequence of books. Waiting for the door to open is different from the pace at which books arrive, and both affect when you can leave with everything you need. A fast handoff cannot erase a long wait outside.

For an application, keep these measurements separate:

These boundaries follow Artificial Analysis's metric definitions. The first reasoning token can arrive before the first answer token, so those clocks need separate names.

Your application's end-to-end latency starts at the user's action and ends at a useful result. For a voice assistant, that might mean audible speech; for an agent, a completed action. Retrieval, tool execution, audio synthesis and interface updates can sit outside the model request. Instrument that wider journey as well.

The underlying stages are prefill, processing the prompt, and decode, generating the continuation. The LLM inference explainer follows that work through the serving system.

Why One Speed Number Misleads

A speed result describes a workload on an endpoint, not an unconditional property of a model. Change the work or the serving conditions and you have changed the comparison.

Prompt length and cached context

A support chat with a short question is different from the same chat carrying a long history and retrieved policy documents. More input changes the work before the answer. Groq identifies input size as a driver of TTFT in its latency guide.

Keep prompt length and cache state in your test record. The KV cache explainer covers the saved attention state used during generation and how prompt reuse relates to caching. A warmed repeated prompt should not silently stand in for a fresh customer conversation.

Reasoning before the answer

A reasoning model may produce intermediate tokens before delivering its answer. Fast generation does not tell you how much reasoning it will do. Ask whether the reported speed includes that stream and whether the first-token timer stops on reasoning or answer text.

For a booking assistant, the useful event is a usable reply or tool call. A visible thinking indicator can acknowledge the request, but it does not complete the booking.

Batch load and concurrency

Batching groups work to improve hardware utilization. Aggregate throughput counts output across requests; per-user speed describes the stream reaching an individual request. Cerebras notes that batching can improve throughput while adding waiting time. Source: Cerebras's throughput explanation.

An operator needs to ask: does each conversation keep moving when other conversations arrive? Measure representative simultaneous traffic, including failed requests and retries. A good result from an otherwise idle endpoint leaves that question unanswered.

Region and delivery

The location of your application and the serving endpoint affects network delay. Groq distinguishes server-side timing from client-experienced latency in its network analysis guidance.

Time from the region where your backend runs. If your app proxies the stream to a browser, add a browser measurement: buffering between those points can make a smooth upstream response appear in bursts. Fixing that path can matter more than changing the model.

Tokens per Second by GPU

A GPU, the accelerator running inference, does not have a single meaningful tokens-per-second rating independent of the model and workload. A hardware comparison needs the model, numerical precision, context, serving configuration and concurrent load beside the result.

Quantization reduces numerical precision to save memory; its speed and quality tradeoffs depend on the implementation. Cerebras documents that distinction in its model-compression explanation. Use the LLM quantization guide when comparing compressed deployments, and preserve an answer-quality check alongside timing.

How Providers State Their Speeds

Treat published speeds as attributed claims with a model and a date attached. The following maker pages were verified on 24 October 2026. These are the makers' figures, not measurements performed for this article.

Cerebras: A Published Speed With a Deployment Caveat

Cerebras, an inference provider, lists OpenAI GPT-OSS 120B at “~3000 tokens/s” in its Developer table. The same page says performance varies by model and configuration and describes Developer access as intended for evaluation rather than production. Source: Cerebras pricing.

The useful question is whether the deployment you can purchase serves your workload with the required responsiveness and capacity. The table alone does not answer it.

Groq Tokens per Second: Read the Model Row

Groq, an inference provider, lists OpenAI GPT-OSS 120B at “500” under “SPEED (T/SEC)” in its supported-model table. That is Groq's published figure for that row. Source: Groq supported models.

These tables do not establish an application-speed ratio between providers. You still need matching prompts, reasoning settings, token accounting, load and measurement boundaries before making that claim.

For the speed-versus-cost decision, compare the price of a successful task and the capacity you need using the Cerebras pricing guide and Groq pricing guide.

Where to Read Independent Measurements

Artificial Analysis, an independent benchmarking service, provides a useful starting point for comparing models and inference providers. Open its provider leaderboard, then inspect the endpoint and workload behind a result.

Its performance methodology currently specifies:

• A default workload of approximately 10,000 input tokens and at least 1,500 answer tokens.
• Results represented by the median over the past 72 hours for the usual workloads.
• A primary testing server in Google Cloud's us-central1-a zone.
• Performance counts normalized with o200k_base, rather than each model's native tokenizer.

That gives you a shared reference, not a promise about your application. Check prompt size, endpoint, reasoning mode and the date before comparing results. Longer prompts can change both first-token delay and output speed.

For reasoning models that hide some reasoning tokens, Artificial Analysis calculates output speed from the last 80% of answer chunks. The simple script below does not reproduce that method or the full benchmark workload. It measures your observed text stream.

Measure Your Own Stream

Measure at the receiving client, and count text tokens rather than streamed events. An event can contain no text or a bundle of text; counting events produces chunks per second.

The following Python example uses the Groq streaming API and tiktoken, a text tokenizer. It reports first-text-chunk delay, request-to-last-text time, stream completion time and an estimate of normalized output tokens per second.

Install the dependencies:

python3 -m pip install groq tiktoken

Set GROQ_API_KEY in your environment. Set GROQ_MODEL to a non-reasoning text model available to your account in Groq's supported-model list, and put a representative application prompt in prompt.txt. Save this as measure_stream.py, then run python3 measure_stream.py.

import os
from pathlib import Path
from time import perf_counter

import tiktoken
from groq import Groq

encoder = tiktoken.get_encoding("o200k_base")
prompt = Path("prompt.txt").read_text(encoding="utf-8")
model = os.environ["GROQ_MODEL"]
parts = []
first = last = None

with Groq(max_retries=0) as client:
    start = perf_counter()
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    for event in stream:
        now = perf_counter()
        if not event.choices:
            continue
        text = event.choices[0].delta.content or ""
        if not text:
            continue
        if first is None:
            first = now
        last = now
        parts.append(text)
    end = perf_counter()

if first is None:
    raise SystemExit("No text received; no text-speed measurement.")

# Tokenize after timing; never count events as tokens.
count = lambda text: len(encoder.encode(text, disallowed_special=()))
total_tokens = count("".join(parts))
post_first_tokens = total_tokens - count(parts[0])
span = last - first

print(f"First text chunk / TTFT estimate (s): {first - start:.3f}")
print(f"Request to last text (s): {last - start:.3f}")
print(f"Stream complete (s): {end - start:.3f}")
print(f"Normalized output tokens: {total_tokens}")
if span > 0 and post_first_tokens > 0:
    print(f"Post-first output tok/s (estimate): {post_first_tokens / span:.1f}")
else:
    print("Output tok/s unavailable: insufficient text delivery intervals.")

The Groq SDK supports disabling automatic retries. This example does so to avoid quietly including retry waits in an otherwise successful sample; a failed request should be recorded as a failure.

The rate calculation excludes the first text chunk from both the token count and the delivery interval. It follows the chunk-based formula in Artificial Analysis's definitions. It also ignores trailing metadata when locating the last text arrival.

Use this example for plain text. A reasoning stream, tool-call arguments or audio needs separate handling for its event types and a definition of the useful first output. For comparisons, preserve the prompt and settings, record output length, and repeat from your deployment region at representative load. Keep typical and slow outcomes; do not promote the fastest sample into a guarantee.

What Calculators and Simulators Can Tell You

A planning aid can illustrate a timing assumption; it cannot discover the delay on your endpoint.

Tokens per Second Simulator

Use a simulator to discuss whether a chat interface should reveal text as it arrives or wait for a completed object. Then validate that design with an observed stream. A constant simulated rate cannot establish whether your real response pauses, arrives in bursts or waits on a tool.

For a completed text response, the accounting is straightforward: request-to-last-text time equals the initial wait plus the first-to-last-text interval. The script measures both intervals instead of inferring them from an advertised rate.

How Many Tokens per Second Is Good?

The useful target comes from the task's waiting points. Set it by answering questions about the experience you are building.

For a voice builder: how long does the caller wait after finishing a sentence before hearing a useful reply? Can speech begin before the entire answer is written? Does the audio buffer run dry? Can the caller interrupt? If silence happens before text starts, a higher decode rate may leave the main problem untouched.

For a chat builder: is the user waiting for the first useful sentence, the complete explanation or a valid structured result? Can the interface display partial output? A support answer and an exportable report have different completion events. Measure the event your interface requires.

For an agent operator: how much of the task is model generation, and how much is tool execution? Does a faster response preserve correct tool arguments and task success? Does the next step wait for the whole output? Improving the wrong interval can leave the workflow's finish time nearly unchanged.

For a provider buyer: can the proposed deployment maintain the needed experience under your traffic, from your region, with your prompts? Is the quoted result typical of that deployment? What happens when requests fail or capacity is busy?

Act now when traces show users waiting on generation or the first useful answer. Wait to migrate providers when the dominant delay has not been isolated. A background job already meeting its completion deadline is largely unaffected by a faster headline unless throughput or successful-task cost changes the decision.

What Is Overhyped About LLM Speed?

The biggest overclaim is that the highest output rate identifies the best application endpoint. Speed does not establish answer quality, valid output or successful tool use.

The wall every fast endpoint hits is the surrounding application. Faster text cannot repair a slow database call, an unnecessarily long reasoning path or a response that fails validation. Equally, a beautiful first-token result can conceal a stream that stalls before the user gets anything useful.

Keep speed in the evaluation, but make a correct, usable result the finish line.

Frequently Asked Questions

Your Next Build Step

For the next iteration, take a representative voice turn, chat prompt or agent task and define what makes its result usable. Instrument that event alongside first output and stream completion. Compare candidate endpoints on the same task, review failures, and optimize the interval that keeps your user waiting.

For practical guides to building useful AI applications, join the newsletter.