What Are Tokens in AI? A Plain-English Guide

Tokens in AI are the small pieces of text a language model reads and generates, such as words, word fragments and punctuation. A bill priced “per million tokens” charges for those pieces, usually at separate input and output rates; token counts also determine what fits in the model's context and affect how long a response takes. For English text, OpenAI estimates about 75 words per 100 tokens, not a fixed conversion. OpenAI's tokenization guide.

What Are Tokens in AI?

A token is a unit of text, not a message or a finished task. A large language model, or LLM, is the kind of AI model that generates language. Its tokenizer, the software that divides text into pieces, prepares that text for the model.

Think of a sentence as a mosaic. You see the whole sentence; the model works with its individual tiles. Those tiles have different sizes: a familiar word may fit on a tile, while another word needs several. The joins do not have to follow syllables or dictionary definitions.

The tokenizer also assigns each piece a numerical identifier. The model processes those identifiers and generates more tokens, which are converted back into readable text. OpenAI's tokenizer documentation shows both the pieces and their identifiers.

For an operator, the useful consequence is simple: a message count is a poor measure of workload. A short instruction accompanied by a document can involve much more text than a long question asked on its own.

How Are AI Tokens Calculated?

Tokens are counted by applying a particular tokenizer to the exact text. Multiplying a word count gives an estimate; running the maker's tokenizer gives a count for that tokenizer.

A counted example from OpenAI's own tool

OpenAI's tiktoken, its open-source tokenizer, splits the exact text tiktoken is great! into 6 tokens with the o200k_base encoding, the named ruleset for converting text into tokens:

t · ikt · oken · is · great · !

The count and split are documented in OpenAI's tokenizer walkthrough. It is a plain-text count, not the total for a chat request.

Notice the leading spaces inside is and great. Spaces can travel with the following word; the exclamation mark is separate here. Even the tool's own name is split into fragments. You cannot reliably count tokens by counting spaces between words.

The example names its encoding deliberately. Change the model's tokenizer and you may change the split. A count is meaningful only alongside the text and tokenizer that produced it.

ChatGPT Token Calculator: Use the Maker's Tools

OpenAI's tokenization guide links to its visual Tokenizer, where you can paste text and inspect its pieces. Use the available encoding or model selection that matches your task; a demonstration under another encoding is not a billing guarantee.

For a complete OpenAI API request, the input-token counting API includes message structure, tools and files that a plain-text count can miss. Ask your builder to use it when estimating an automation.

Other makers provide their own counting tools:

• Anthropic Claude: the token-counting endpoint estimates input usage for the selected model, including messages, standing instructions, tools, images and PDFs.
• Google Gemini: the count_tokens tool counts input before generation, while the response's usage information reports what was used.

These tools count according to each maker's rules. For an automation using an API, the connection between an app and the AI model, ask its builder to count the actual model and request format.

How many words is a token?

For English text, OpenAI gives these approximations: a token is about 4 characters or three-quarters of a word, and 100 tokens are about 75 words. These are the maker's estimates, not fixed conversions. OpenAI's documentation gives the shortcut specifically for English; the exact count depends on the text and encoding.

Use that shortcut to judge whether a draft is small or large. For a multilingual campaign, a product catalog or a document containing code, count the material itself before promising a capacity or budget.

Input and Output Tokens: What You Send and What Comes Back

Input tokens are what the model receives; output tokens are what it generates. The input can include much more than the latest sentence you typed.

Imagine a customer-support workflow. Your visible instruction says to draft a reply. The application may also send its standing instructions, the customer's previous messages and the relevant returns policy. Those supplied materials belong on the input side. The draft reply belongs on the output side.

Anthropic's counting documentation covers structured inputs such as instructions, messages and tools. Google's multi-turn examples show why previous conversation turns can contribute to subsequent usage.

For a marketer, this means attaching the entire brand archive to every small copy request deserves scrutiny. For a founder, it means asking the builder to measure the complete request rather than estimating from the visible chat bubble.

There is another distinction on the output side. OpenAI's reasoning-model documentation says internal reasoning tokens count as output usage even though they are not shown as answer text. A brief visible answer can therefore have a larger output count than its displayed words suggest.

Why Are Output Tokens Often Priced Higher?

Reading input and generating output are different kinds of work, so providers price them separately. Generating the response is often the slowest part of a request, according to OpenAI's latency guide. That helps explain why long answers deserve attention when reviewing both cost and speed.

There is no universal input-to-output price ratio. The practical implication is that a workflow which reads documents and returns short labels has a different usage mix from one that writes long reports. Compare those workloads using their separate input and output totals.

Since October 26, 2026, GPT-6.1 Terra has offered another option: $1 per million input tokens and $5 per million output tokens at standard short-context API rates. Google's October 28 announcement gives Gemini 4 Carbon introductory rates of $2 input / $10 output, rising to $4 / $20 after the introductory period, all per million tokens; wider access is still pending. Verified October 31, 2026: budget with the model and rate available to your account, not an announced future option.

For ordinary uncached token usage, calculate input tokens ÷ 1,000,000 × input rate, then add output tokens ÷ 1,000,000 × output rate. “Per million” is the pricing unit, not a requirement to consume that much. Paid tools, storage and special processing tiers can add separate charges.

Worked Example: A 10-Page Document in Tokens and Dollars

Page count alone cannot tell you a document's text-token count. A dense contract and a slide deck can have the same page count and very different amounts of text. For a PDF, the model's document-processing rules can give you a page-based estimate.

Google's document-tokenization guidance gives a baseline of 258 tokens per PDF page. Using that baseline with Gemini 2.5 Flash, Google's standard pricing, verified October 31, 2026, is $0.30 per million input tokens for text and images:

That is the document-page input subtotal, not the whole summary bill. Your instruction, any additional billable input and the generated answer add to it. This model's standard output rate is $2.50 per million tokens, including thinking tokens. Use this as a page-processing estimate, then check the actual request's usage before projecting monthly volume. Other models and document-processing settings can count pages differently; count the complete request for the model you use.

An automation's bill accumulates across its API requests. Revisions and retries add work too. Cached input, where a provider reuses previously processed prompt material, can have different billing rules. The mechanics belong in our KV-cache and prompt-caching explainer. For model shopping, use the cheapest AI API comparison; this example is here to explain the unit.

An AI Token Limit Can Mean Different Things

Find out which limit you hit before changing plans. A request that is too large, a burst of requests that is too fast and an exhausted subscription allowance need different fixes.

A context window is the token capacity available to a request: the material you supply and the output the model generates must fit. A maximum-output limit separately caps how much it can generate. Keep room for the answer and any internal reasoning as well as the material you supply. The broader explanation is in our guide to context size and context engineering.

An API rate limit restricts how quickly you can send requests or tokens. A spend limit concerns charges. Neither tells you how much material fits in a particular request. OpenAI explains request and usage restrictions in its rate-limit documentation.

A chat-plan allowance governs usage under that product's rules. Anthropic says Claude usage varies with conversation length and complexity, selected model, features and effort. Its usage-and-length guide also explains that usage allowances and conversation-length limits are separate.

For an operator, the consequence is practical: buying more usage does not necessarily make an oversized document fit. Starting a fresh conversation may reduce the material being carried forward, but it does not erase usage already consumed.

How to Buy AI Tokens

You usually pay for access or metered usage through the product you use. Check whether its offer is a subscription, API billing or an app-specific credit balance before adding money.

For example, ChatGPT subscriptions and API usage have separate billing. OpenAI says eligible subscriptions may provide API credits for a linked organization, but API usage is still metered separately. Other products can bundle usage differently; read the terms of the product charging you.

How Tokens Affect AI Speed

Longer generated answers usually mean a longer wait. Output tokens per second measures how quickly the answer arrives once generation is under way. The wait before it starts, called time to first token, is a separate part of the experience. Anthropic's terminology guide explains that initial delay.

For a founder reviewing a slow support assistant, ask whether the delay happens before the answer starts or while it writes. OpenAI's latency guidance recommends reducing unnecessary output and using streaming, which displays text as it arrives. A concise reply can save both output charges and waiting time; streaming makes progress visible without removing the work. Keep the detail the customer needs, and measure the completed task rather than judging speed from a headline token rate.

What's Overhyped About Token Counts?

A larger token number is not an answer-quality score. Capacity tells you something about what can fit. It does not establish whether the answer is correct, useful or worth the spend.

Likewise, a lower token rate does not settle which model is cheaper for your job. Tokenizers can divide the same material differently, and workflows can generate different amounts of output and reasoning. Compare the cost of completing an acceptable result, using the cheapest LLM API guide to compare current rates.

Do not turn token reduction into a contest to write the shortest possible prompt. Removing irrelevant attachments is sensible. Removing the policy exception that makes a customer reply accurate is a false saving. Keep the information that changes the answer and remove material that does not serve the task.

What to Do Differently Next Week

Act now if you own an automated workflow or repeatedly hit a limit. If you use a chat app occasionally and stay within its allowance, you do not need to count every sentence.

1. 1

   For operators: identify the meter

   Read the usage screen and the limit message. Record whether the constraint is request length, request rate, spend or a plan allowance. Choose the fix for that constraint.

2. 2

   For marketers: count a representative task

   Take an actual brief and the supporting material you normally send. Inspect it with the maker's counting tool, then ask for the output format the job needs. Keep essential brand and factual instructions.

3. 3

   For founders and buyers: request the usage breakdown

   Ask your builder or vendor for input, output, cached usage and any separate tool charges from a representative completed task. Use that breakdown to forecast your own volume before upgrading or switching models.

Questions About AI Tokens

What is a token in AI?

A token is a piece of text a language model processes. It can be a word fragment, a whole word or punctuation; it is not automatically a whole word.

What does tokenization mean in the context of AI?

Tokenization is the process of splitting input into the pieces the model uses and representing those pieces in its vocabulary. The tokenizer's rules determine the split.

Who are the big 4 of AI?

That phrase concerns company rankings, not a token standard. For counting and billing, identify the maker and exact model behind your tool; a list of prominent companies will not determine its token usage.

How much is 1 million tokens in ChatGPT?

There is no universal ChatGPT conversion into a dollar amount. ChatGPT product billing and OpenAI API billing are separate. For an API calculation, identify the model and distinguish input from output; for a chat plan, check that plan's usage rules.

How much is an AI token worth?

It has no fixed monetary value. Its usage charge depends on the provider, model and billing category. A token is a processing unit, not a transferable currency.

Who pays for AI tokens?

For a directly billed API workflow, the account running the requests pays the provider. In a packaged app, the app's agreement determines whether you pay a subscription, usage charges or both. Ask which charges your quoted price includes.

How much is 1000 tokens in AI?

As a charge, it depends on the model and whether the tokens are input, output or cached. As an amount of text, it varies with the tokenizer and language. Use the maker's tokenizer instead of treating it as a fixed word allowance.

Can you explain how ChatGPT tokens work?

Text is divided into tokens for processing and generation. The text you type and the text you receive are not necessarily the entire usage: supplied context and, for supported reasoning models, internal reasoning can contribute too.

What AI has the most tokens?

First decide whether you mean the largest context window, longest possible output or most included usage. Those measure different things. Compare the current documentation for the specific models or plans on that basis.

Can you use AI for free?

Yes. For example, Claude offers a free plan, with limits. Free access does not mean unlimited usage or an unlimited context window.

For practical explanations of AI tools and the decisions behind them, join the newsletter.