What Are Tokens in AI? A Plain-English Guide

Tokens in AI are the small pieces of text a language model reads and generates, such as words, word fragments and punctuation. OpenAI's English rule of thumb is about three-quarters of a word per token, or 75 words per 100 tokens. They matter because they help determine usage charges and how much text a model can handle. Source: OpenAI's token guide.

What Are Tokens in AI?

A token is a unit of text, not a message or a finished task. A large language model, or LLM, is the kind of AI model that generates language. Its tokenizer, the software that divides text into pieces, prepares that text for the model.

Think of a sentence as a mosaic. You see the whole sentence; the model works with its individual tiles. Those tiles have different sizes: a familiar word may fit on a tile, while another word needs several. The joins do not have to follow syllables or dictionary definitions.

The tokenizer also assigns each piece a numerical identifier. The model processes those identifiers and generates more tokens, which are converted back into readable text. OpenAI's tokenizer documentation shows both the pieces and their identifiers.

For an operator, the useful consequence is simple: a message count is a poor measure of workload. A short instruction accompanied by a document can involve much more text than a long question asked on its own.

How Are AI Tokens Calculated?

Tokens are counted by applying a particular tokenizer to the exact text. Multiplying a word count gives an estimate; running the maker's tokenizer gives a count for that tokenizer.

A counted example from OpenAI's own tool

OpenAI's tiktoken, its open-source tokenizer, splits the exact text tiktoken is great! into 6 tokens with the cl100k_base encoding, the named ruleset for converting text into tokens:

t · ik · token · is · great · !

This count was reproduced with OpenAI's tool on 15 October 2026. The sentence and split also appear in OpenAI's tokenizer walkthrough. It is a plain-text count, not the total for a chat request.

Notice the leading spaces inside is and great. Spaces can travel with the following word; the exclamation mark is separate here. Even the tool's own name is split into fragments. You cannot reliably count tokens by counting spaces between words.

The example names its encoding deliberately. Change the model's tokenizer and you may change the split. A count is meaningful only alongside the text and tokenizer that produced it.

ChatGPT Token Calculator: Use the Maker's Tools

OpenAI's visual Tokenizer lets you paste text and inspect its pieces. Use the available encoding or model selection that matches your task; a demonstration under another encoding is not a billing guarantee.

Other makers provide their own counting tools:

• Anthropic Claude: the token-counting endpoint counts supported message inputs for the selected model. Its documentation identifies request features it cannot count.
• Google Gemini: the count_tokens tool counts input before generation, while the response's usage information reports what was used.

These tools count according to each maker's rules. For an automation using an API, the connection between an app and the AI model, ask its builder to count the actual model and request format.

How many words is a token?

For English text, OpenAI gives these approximations: a token is about 4 characters or three-quarters of a word, and 100 tokens are about 75 words. These are the maker's estimates, not fixed conversions. OpenAI's help article explicitly notes that language and text structure change the relationship.

Use that shortcut to judge whether a draft is small or large. For a multilingual campaign, a product catalog or a document containing code, count the material itself before promising a capacity or budget.

Input and Output Tokens: What You Send and What Comes Back

Input tokens are what the model receives; output tokens are what it generates. The input can include much more than the latest sentence you typed.

Imagine a customer-support workflow. Your visible instruction says to draft a reply. The application may also send its standing instructions, the customer's previous messages and the relevant returns policy. Those supplied materials belong on the input side. The draft reply belongs on the output side.

Anthropic's counting documentation covers structured inputs such as instructions, messages and tools. Google's multi-turn examples show why previous conversation turns can contribute to subsequent usage.

For a marketer, this means attaching the entire brand archive to every small copy request deserves scrutiny. For a founder, it means asking the builder to measure the complete request rather than estimating from the visible chat bubble.

There is another distinction on the output side. OpenAI's reasoning-model documentation says internal reasoning tokens count as output usage even though they are not shown as answer text. A brief visible answer can therefore have a larger output count than its displayed words suggest.

Why Are Output Tokens Often Priced Higher?

Reading an existing prompt and generating a continuation put different demands on the computing system. The full input is available up front, so much of its processing can happen in parallel. Conventional output generation proceeds sequentially, with later tokens depending on earlier ones. NVIDIA explains this distinction in its technical guide to input processing and generation.

That helps explain separate input and output rates. It does not establish a universal price ratio: providers set their own rates, and model design and serving techniques differ.

The practical implication is that a workflow which reads documents and returns short labels has a different usage mix from one that writes long reports. Compare those workloads using their separate input and output totals.

Worked monthly example: Anthropic currently lists Claude Sonnet 5.5 at $2 per million input tokens and $10 per million output tokens at standard API rates. If a month's aggregate usage is 1 million uncached input tokens plus 1 million output tokens, our arithmetic gives $2 + $10 = $12 in model token charges. This illustrative subtotal assumes standard global processing, no caching or batch discounts, and excludes paid tools and taxes. Maker pricing, verified 15 October 2026.

That is a usage subtotal, not a subscription price or a prediction of your monthly volume. An automation's count accumulates across its API requests. Revisions and retries add work too.

Cached input, where a provider reuses previously processed prompt material, can have different billing rules. The mechanics belong in our KV-cache and prompt-caching explainer. For model shopping, use the cheapest AI API comparison; this example is here to explain the unit.

An AI Token Limit Can Mean Different Things

Find out which limit you hit before changing plans. A request that is too large, a burst of requests that is too fast and an exhausted subscription allowance need different fixes.

A context limit concerns how much material the model can work with in a request. A maximum-output limit concerns how much it can generate. Keep room for the answer as well as the material you supply. The broader explanation is in our guide to context size and context engineering.

An API rate limit restricts how quickly you can send requests or tokens. A spend limit concerns charges. Neither tells you how much material fits in a particular request. OpenAI distinguishes these limits in its token guide.

A chat-plan allowance governs usage under that product's rules. Anthropic says Claude usage varies with conversation length and complexity, selected model, features and effort. Its usage-and-length guide also explains that usage allowances and conversation-length limits are separate.

For an operator, the consequence is practical: buying more usage does not necessarily make an oversized document fit. Starting a fresh conversation may reduce the material being carried forward, but it does not erase usage already consumed.

How to Buy AI Tokens

You usually pay for access or metered usage through the product you use. Check whether its offer is a subscription, API billing or an app-specific credit balance before adding money.

For example, OpenAI states that ChatGPT subscriptions and API usage have separate billing. Paying for the chat product does not make the API's token charges disappear. Other products can bundle usage differently; read the terms of the product charging you.

What's Overhyped About Token Counts?

A larger token number is not an answer-quality score. Capacity tells you something about what can fit. It does not establish whether the answer is correct, useful or worth the spend.

Likewise, a lower token rate does not settle which model is cheaper for your job. Tokenizers can divide the same material differently, and workflows can generate different amounts of output and reasoning. Compare the cost of completing an acceptable result.

Do not turn token reduction into a contest to write the shortest possible prompt. Removing irrelevant attachments is sensible. Removing the policy exception that makes a customer reply accurate is a false saving. Keep the information that changes the answer and remove material that does not serve the task.

What to Do Differently Next Week

Act now if you own an automated workflow or repeatedly hit a limit. If you use a chat app occasionally and stay within its allowance, you do not need to count every sentence.

1. 1

   For operators: identify the meter

   Read the usage screen and the limit message. Record whether the constraint is request length, request rate, spend or a plan allowance. Choose the fix for that constraint.

2. 2

   For marketers: count a representative task

   Take an actual brief and the supporting material you normally send. Inspect it with the maker's counting tool, then ask for the output format the job needs. Keep essential brand and factual instructions.

3. 3

   For founders and buyers: request the usage breakdown

   Ask your builder or vendor for input, output, cached usage and any separate tool charges from a representative completed task. Use that breakdown to forecast your own volume before upgrading or switching models.

Questions About AI Tokens

What is a token in AI?

A token is a piece of text a language model processes. It can be a word fragment, a whole word or punctuation; it is not automatically a whole word.

What does tokenization mean in the context of AI?

Tokenization is the process of splitting input into the pieces the model uses and representing those pieces in its vocabulary. The tokenizer's rules determine the split.

Who are the big 4 of AI?

That phrase concerns company rankings, not a token standard. For counting and billing, identify the maker and exact model behind your tool; a list of prominent companies will not determine its token usage.

How much is 1 million tokens in ChatGPT?

There is no universal ChatGPT conversion into a dollar amount. ChatGPT product billing and OpenAI API billing are separate. For an API calculation, identify the model and distinguish input from output; for a chat plan, check that plan's usage rules.

How much is an AI token worth?

It has no fixed monetary value. Its usage charge depends on the provider, model and billing category. A token is a processing unit, not a transferable currency.

Who pays for AI tokens?

For a directly billed API workflow, the account running the requests pays the provider. In a packaged app, the app's agreement determines whether you pay a subscription, usage charges or both. Ask which charges your quoted price includes.

How much is 1000 tokens in AI?

As a charge, it depends on the model and whether the tokens are input, output or cached. As an amount of text, it varies with the tokenizer and language. Use the maker's tokenizer instead of treating it as a fixed word allowance.

Can you explain how ChatGPT tokens work?

Text is divided into tokens for processing and generation. The text you type and the text you receive are not necessarily the entire usage: supplied context and, for supported reasoning models, internal reasoning can contribute too.

What AI has the most tokens?

First decide whether you mean the largest context window, longest possible output or most included usage. Those measure different things. Compare the current documentation for the specific models or plans on that basis.

Can you use AI for free?

Yes. For example, Claude offers a free plan, with limits. Free access does not mean unlimited usage or an unlimited context window.

For practical explanations of AI tools and the decisions behind them, join the newsletter.