Chain of Thought Prompting: When to Use It

Use chain of thought prompting when a task benefits from explicit intermediate work, but do not make “think step by step” your default instruction for a reasoning model. The technique introduced in 2022 used worked examples to guide a model toward an answer; today's reasoning models can do internal work without printing it. For business tasks, specify the evidence and result you need, then tune thinking effort only when the task warrants it.

What Is Chain of Thought Prompting?

Chain-of-thought prompting gives a language model intermediate steps to follow, or asks it to produce them, instead of supplying only the final answer. A large language model, or LLM, generates text from the instructions and context you provide. CoT is a way to structure that interaction.

Jason Wei and coauthors introduced the named technique in Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, first submitted on 28 January 2022. Their prompt examples paired a problem with a short explanation and its answer. The model then continued that pattern for a new problem. This was prompting an existing model, not retraining it for each task. Original paper.

Think of an analyst learning a stock reconciliation. Showing only the closing balance hides the method; showing opening stock, shipments, receipts and the resulting balance demonstrates the calculation. That is the intuition behind the original technique.

A modern reasoning model adds a distinction: the analyst's scratch work and the memo delivered to a manager are separate things. You can give the model more room to work internally while still requesting a short final answer.

What Is the Purpose of Chain of Thought Prompting?

The purpose is to help with a problem whose intermediate operations matter. For a model that benefits from explicit demonstrations, a worked example can show which quantities to combine or which rule to apply.

For an operator, the useful question is whether the result becomes more correct and easier to check. A longer explanation alone does not establish either. Treat the prompt as a candidate improvement to evaluate on your own task.

Chain of Thought Prompting Examples for Business Tasks

Use a worked example to teach a method; use an output requirement to specify what a reviewer needs to see. Those goals can overlap, but they are different reasons to request steps.

A Chain of Thought Prompting Example: Before and After

Here is an illustrative stock-reconciliation prompt, adapted to business work rather than copied from the paper. The arithmetic is supplied for explanation; these are not outputs from a model test.

Before: an answer-only demonstration

Example question: Start with 12 units, ship 5 and receive 3. How many remain?

Example answer: 10 units.

New question: Start with 18 units, ship 7 and receive 4. How many remain?

After: the same demonstration with intermediate work

Example question: Start with 12 units, ship 5 and receive 3. How many remain?

Example answer: Remaining stock = opening stock − shipments + receipts = 12 − 5 + 3 = 10 units.

New question: Start with 18 units, ship 7 and receive 4. How many remain?

The change is the method inside the example. The new question has the same correct answer in both cases: 15 units. The prompt does not prove that a model will answer correctly; it demonstrates how few-shot CoT works. “Few-shot” means giving examples in the prompt, and a single demonstration is the smallest version of that pattern.

Extraction: Ask for the Field and Its Evidence

For an accounts-payable operator extracting an invoice number, start with a direct instruction:

Return the invoice number exactly as written and the source line containing it. If it is absent, return null. Do not infer a number from other identifiers.

There is little value in an essay explaining how the model noticed a label. The source line lets a reviewer check the extraction. If the task expands to reconciling inconsistent purchase orders and invoices, it becomes a candidate for more reasoning effort.

Decisions: Ask for the Applied Rule

For a support operator reviewing a refund request, specify the decision artifact:

Apply only the supplied refund policy. Return the recommendation, the applicable policy clause, the customer facts supporting it and any missing fact that prevents a decision. Keep the rationale concise.

That prompt defines useful evidence without requesting a transcript of private reasoning. Keep reusable policy and output instructions in your system prompt, with the case-specific documents clearly separated.

Zero-Shot Chain-of-Thought Prompting

Zero-shot CoT asks for intermediate reasoning without providing a worked example. Takeshi Kojima and coauthors studied this in Large Language Models are Zero-Shot Reasoners, first submitted on 24 May 2022, using the phrase “Let's think step by step.” Zero-shot CoT paper.

Applied to the stock task, the prompt becomes:

Start with 18 units, ship 7 and receive 4. How many remain? Let's think step by step.

The phrase supplies a general instruction rather than demonstrating a particular method. It is useful to understand historically, but it is not a universal switch that improves every current model.

The variants also describe different mechanisms. Few-shot CoT supplies worked demonstrations. Zero-shot CoT supplies an instruction without demonstrations. Auto-CoT uses a model to construct demonstrations from diverse questions; it is not another name for a reasoning model thinking internally. Auto-CoT paper.

For business work, start with a clear task and inspect failures. Add examples when the missing piece is a method, a boundary case or the required output format.

What Reasoning Models Change

With native reasoning, your main job is to define the task, constraints and deliverable. The model's reasoning control determines how much internal work it can apply. Instructions about the final explanation determine what the user receives.

The guidance below was verified against OpenAI, Anthropic and Google's live documentation on 18 October 2026. Controls differ by model and API, so there is no portable “thinking budget” setting you can paste into every provider.

OpenAI: Keep Reasoning Prompts Direct

OpenAI's reasoning-model guidance says step-by-step prompting is unnecessary because these models reason internally, and can sometimes hinder performance. Start with the goal, relevant context, constraints and a clear output format. Reasoning best practices.

That advice does not describe every OpenAI model. The GPT-4.1 prompting guide identifies it as a non-reasoning model and describes explicit stepwise prompting as potentially useful, with extra output and latency. Select the prompting approach for the model you are actually using. GPT-4.1 prompting guide.

Anthropic: Use Thinking and Effort Before Prescribing Every Step

Anthropic's Claude prompting guide prefers broad thinking guidance over a hand-written sequence of reasoning steps. Worked examples can still teach an approach. Manual CoT is a fallback when thinking is off on applicable models; some newer models may decline requests to write out their internal reasoning. Claude prompting best practices.

The practical order is to choose an effort level for your workload, then add prompt steering if the model's behavior still needs adjustment. A simple request can receive a direct response, and an effort setting does not guarantee a visible thinking block. Thinking steering.

Google: Internal Thinking Usually Does Not Need a Printed Plan

Google's Gemini prompting guide says the Gemini 2.5 and Gemini 3 series generate internal thinking, so returned outlines or detailed reasoning steps are generally unnecessary. It also says a simple instruction to think harder can help demanding problems, with additional thinking tokens. This is task-specific guidance, not a promise that adding the phrase improves every answer. Gemini prompt design strategies.

For a field lookup, Google's thinking guide recommends minimal or low thinking where supported. Reserve greater effort for complex tasks such as multistep planning. Gemini thinking.

Reasoning Effort Versus a Thinking Budget

An effort level steers how deeply a model works. A token budget specifies an allowance for thinking on configurations that support it. Neither tells the model which evidence belongs in the final answer.

These are field names to locate in your integration, not a shared configuration format. Check the relevant OpenAI reasoning reference, Claude thinking guide and Gemini Generate Content reference before changing them.

Billing: OpenAI states that reasoning tokens are billed as output tokens even though the raw reasoning tokens are not visible through the API. Provider billing statement.

Tokens are the units the model processes and generates; what tokens mean in AI explains the distinction between visible text and token usage. A short answer can still involve substantial internal work.

When to Ask for Visible Steps

Ask for visible steps when those steps are part of the useful deliverable. A finance operator may need the formula and substituted inputs. A support reviewer may need the policy clause and matching facts. A business analyst may need assumptions, source references and the calculation behind a recommendation.

Make the request concrete:

Return the recommendation, supporting source references, key assumptions and a concise explanation. Show any calculations needed to reproduce the result. Flag missing evidence.

That is an explanation a person can assess. It does not demand every internal thought, failed approach or private intermediate state.

Provider thinking summaries are another output type. OpenAI exposes supported reasoning summaries rather than raw reasoning tokens; Anthropic describes Claude's thinking text as a summary rather than the raw chain of thought. Treat these summaries as diagnostic context, not a complete account of the computation. OpenAI summaries, Claude thinking output.

For a builder, define fields your application can validate. For an operator, define evidence a reviewer can check. If the decision depends on a current balance or order status, use a tool to retrieve that fact. Function calling connects the model to those external operations; asking it to reason longer does not provide the missing record.

When Chain of Thought Hurts

Skip elaborate reasoning prompts when the job is simple retrieval or fixed-format extraction. Returning a known account ID should not require a narrated argument. Extra prose increases the material your application must parse and your reviewer must read.

Latency is the next limit. Explicit reasoning text adds output, while higher native effort can delay the useful answer. OpenAI documents the output-and-latency tradeoff for GPT-4.1, and Google recommends lighter thinking for simple tasks. The practical rule is to start with a direct prompt and increase complexity only when your evaluations justify it. GPT-4.1 guide, Gemini thinking guidance.

Missing information is a different failure. If the relevant policy was never supplied, a longer reasoning request is the wrong repair. Retrieve the policy and identify its version. If the input is too large, fix what enters the context window, the model's available working space, before experimenting with more thinking.

A rigid method can also be wrong for the task. A stock example that handles receipts and shipments says nothing about returns, damage or reserved inventory. Keep examples representative and allow the model to report a missing rule.

What the Original Numbers Do and Do Not Prove

The original benchmark establishes a historical result on a defined task, not a promised business accuracy gain. Wei et al.'s NeurIPS 2022 paper reports 17.9% accuracy with standard prompting and 56.9% with chain-of-thought prompting for PaLM 540B on GSM8K, a benchmark of math word problems. The setup used eight worked demonstrations. These figures are from Table 1 in the paper revision dated 10 January 2023. Paper and results.

Those numbers do not estimate the gain for invoice extraction, refund recommendations or a current reasoning model. The model, benchmark and prompting setup all matter. Use them to understand why the technique attracted attention, then measure the task you intend to ship.

What Is Overhyped About Chain of Thought?

A convincing explanation is not a complete audit trail. In research published on 3 April 2025, Anthropic found cases where models used supplied hints but omitted that influence from their explanations. The experiments were controlled scenarios with particular models and tasks, not a measurement of every business workflow. They still show why a rationale should not be treated as proof of how an answer was produced. Anthropic's faithfulness research.

For an auditable decision, preserve the supplied documents, retrieved records, tool results, applied rules and final output. Check citations against the source and recompute arithmetic where possible. A model-written explanation can help a reviewer understand the recommendation, but it cannot replace those records.

The other overstatement is that reasoning models make prompt design irrelevant. They change what you need to specify. The business objective, evidence boundary, exception handling and output format still belong to you.

What to Change in Your Workflow This Week

Act now if a blanket “think step by step” instruction is attached to every request. Builders should test removing it from routine extraction and compare direct task prompts on the same cases. Operators should replace requests for exhaustive reasoning with a defined review artifact: evidence, calculations, assumptions and a concise recommendation.

Keep a working prompt while you test its replacement. A non-reasoning model using useful demonstrations is not automatically obsolete. A migration to a reasoning model is a reason to re-evaluate, not evidence that the old workflow should be deleted.

1. 1

   Choose a task with a checkable answer

   Use a recurring job such as invoice-field extraction or stock reconciliation. Write down what counts as correct, what should return missing information and what your software must be able to parse.

2. 2

   Compare one change at a time

   Hold the model and input cases fixed while comparing a direct prompt with a worked-example prompt. Then, on a reasoning model, hold the prompt fixed while changing a supported effort level. Record answer correctness, missing-evidence behavior, completion time and token usage.

3. 3

   Keep only changes that earn their complexity

   Prefer deterministic checks for arithmetic and exact fields. Where judgment is needed, use a rubric and human review; an LLM as a judge can help scale evaluation, but its grades also need calibration. Ship the version that meets your quality and response-time requirements.

The Monday move: take one prompt used in a recurring business task, specify the evidence its reviewer needs and run the comparison above. Keep the simplest configuration that passes your checks.

Questions About Chain-of-Thought Prompting

What is the chain-of-thought prompting?

It is prompting a language model with intermediate steps or worked demonstrations before the final answer. The original few-shot technique demonstrates a method; zero-shot CoT asks for steps without providing examples.

What is an example of a chain of thought?

In AI prompting, a worked stock calculation is an example: opening stock minus shipments plus receipts gives closing stock. Providing that calculation alongside a solved example demonstrates how to approach a similar question.

Can you give me an example of a chain of thought prompt?

“Start with 18 units, ship 7 and receive 4. How many remain? Let's think step by step.” That illustrates zero-shot CoT. With a native reasoning model, you can instead request the closing stock and a reproducible calculation as the final deliverable.

Is chain-of-thought prompting still relevant?

Yes, particularly for models that benefit from explicit demonstrations and tasks where a worked explanation is useful. Reasoning models change the default: start with clear instructions and native reasoning controls, then add examples or explanation requirements where they help.

Who invented Chain of Thought prompting?

Jason Wei and coauthors introduced the named technique in their 2022 paper, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Their work combined intermediate natural-language explanations with examples in the prompt; it built on earlier research into rationales and prompting.

What is the main benefit of using Chain of Thought prompting?

It can help a suitable model handle a problem through intermediate operations rather than jumping directly to an answer. Whether that improves your workflow depends on the model and task; a longer explanation is not itself a quality measure.

What is the main difference between prompt chaining and Chain of Thought prompting?

Prompt chaining divides a workflow across multiple model calls, passing one result into the next. Chain-of-thought prompting concerns intermediate steps within a prompt demonstration or response. A workflow can use both, but they are separate design choices.

For practical guides to building useful AI workflows, join the newsletter.