Context Rot: Why Long Chats Degrade and How to Fix It

Context rot can make an agent less accurate even when its input fits the advertised context window. Chroma's study of 18 language models found that reliability changed with input length. The practical response is to preserve the evidence for the next action while reducing irrelevant history.

What Is Context Rot in AI?

Context rot is declining reliability as the information supplied to a language model grows. It can show up as missed facts, incorrect associations or answers that stop respecting an established decision. A long conversation may still be accepted by the model while becoming harder for it to use accurately.

The context window is the model's available working space for a request. Its contents can include instructions, conversation history, documents and tool results. A token is a unit of content the model processes, often a fragment of a word. Our context-window guide explains what consumes that space.

Think of an incident room where the wall holds the current plan alongside abandoned hypotheses, superseded timelines and old logs. Enlarging the wall lets you keep more material. It does not tell the next person which decision still governs the work.

The distinction matters when you debug an agent. If the required fact was removed before the model received the request, investigate truncation or retrieval. If the fact was present but the answer missed or misused it, investigate how the model handled the supplied context. Neither symptom, by itself, proves a length-related problem.

Documentation checked October 24, 2026: the maker guidance below was verified against Anthropic's live documentation. The workflow examples are proposed designs, not reported production results.

The Context Rot Paper: What the Evidence Shows

Chroma's report, published July 14, 2025, found that harder semantic matches and relevant-looking distractors made longer-input retrieval less reliable. Its controlled tasks are useful evidence, not a benchmark of your agent. Original study.

In its cleaned LongMemEval conversation benchmark, Chroma compared 306 prompts averaging about 113,000 tokens with focused versions averaging about 300 tokens. Focused inputs scored better across the evaluated models. Crucially, those short inputs came from labeled relevant passages and manual adjustments, not an automatically evaluated retrieval system. Study methods and results.

Chroma Context Rot Findings Have a Boundary

The report does not establish the underlying mechanism or a universal safe token count. Its repeated-text experiment also grows the output with the input, unlike its short-answer retrieval tests. Research limitations and task descriptions.

For a builder, the useful question is whether your system can find and preserve the necessary evidence. A chart comparing carefully selected evidence with a full history does not tell you whether your search index, filters or summarizer will select that evidence correctly. That selection needs its own evaluation.

Why Longer Context Can Be Harder to Use

Relevant material must be distinguishable from everything around it. An old deployment plan can resemble the current plan more closely than an unrelated document does. The risk is not just failing to find a sentence; it is applying the wrong sentence to the current situation.

Anthropic's context-engineering guidance describes architectural constraints and training on predominantly shorter sequences as contributors to long-context difficulty. It describes a gradual loss of precision, rather than a single boundary where capability disappears. These are explanations to reason with, not proof of a complete causal account.

Consider a support agent whose history contains an initial diagnosis, a failed workaround and a later confirmed fix. If a subsequent tool call returns an older version of the troubleshooting guide, the next answer must reconcile both chronology and authority. More text has made the decision more complicated even though the user's question is unchanged.

Make that reconciliation explicit: identify the current state, mark the earlier diagnosis as rejected, and retain the source that supports the accepted fix. The aim is sufficient evidence with clear status, not the shortest prompt you can manufacture.

What Builders and Operators Should Check First

Compare the same decision with different context before blaming the model. For builders, this means capturing the actual request assembled by the application. The visible chat transcript may omit system instructions, retrieval results or tool responses that reached the model.

For operators, identify a concrete missed requirement or wrong action. “The chat feels worse” is hard to investigate; “it recommended the workaround already rejected in the incident record” gives you something to replay.

Use this diagnostic sequence on a saved example:

1. 1

   Freeze the decision

   Keep the question, model configuration and required outcome fixed. Save the original input and authoritative evidence. Replay without executing external actions, so the comparison cannot change the underlying case.

2. 2

   Check whether the evidence arrived

   Find the necessary passage in the actual request. If it is absent, fix retrieval, truncation or state loading first. If an obsolete source arrived instead, fix source selection and freshness.

3. 3

   Build a focused comparison

   Retain the same task instructions and all evidence needed to answer, while removing unrelated history and obsolete tool results. Do not include the expected answer or simplify the task. Compare correctness, source use and constraint compliance across representative replays.

4. 4

   Choose the repair from the failure

   If focused input helps, investigate context selection and organization. If both versions fail, examine the task, instructions, evidence quality and model capability. If failure starts after summarization, inspect what the summary lost.

This comparison is a diagnostic, not proof that length alone caused the error: removing material also changes its composition. For a stricter length check, keep the relevant evidence and its placement controlled while adding unrelated material. Do not generalize from a lucky rerun.

Teams should act now when growing history repeatedly accompanies preventable mistakes. A short, bounded workflow that meets its correctness checks can keep its simpler design. Add orchestration when the failure gives it a job.

How to Reduce Context Rot in an Agent

Match the intervention to the material causing the problem. Compaction manages accumulated history; retrieval selects evidence; subagents contain exploration; ordering clarifies the task; concise tool responses reduce unnecessary input. These are complementary controls, with different ways to fail.

Compact the History, Preserve the Working State

Compaction replaces older conversation material with a shorter continuation record. Anthropic's live compaction documentation explains why keeping active context small matters: “response quality degrades as a conversation grows.” Its API supports both application-requested compaction and compaction triggered by a configured token threshold.

Design the handoff around what the next action requires. For the incident example, retain the current objective, confirmed cause, rejected workaround and reason for rejecting it, remaining uncertainty, source locations and next action. Preserve exact identifiers or text when correctness depends on them.

A weak handoff says, “Investigated the issue and made progress.” A useful handoff says, “The workaround was rejected because it breaks the required behavior; the accepted fix is recorded in the incident document; verification is still pending.” The second supports continuation without retelling the investigation.

The wall: a summary can discard a detail that becomes important later. Keep the original record accessible, and evaluate whether the resumed agent can still answer the questions that require it. Saving durable state and choosing what to reload are separate jobs, covered in agent memory explained.

Retrieve Evidence When the Task Needs It

Keep references to large sources and load the relevant portions on demand. Anthropic describes this as just-in-time context: the agent can retain file paths, queries or links, then fetch material as needed. Context retrieval guidance.

For an incident agent, start with the current incident record and the applicable service documentation. Fetch an older incident when the diagnosis calls for it. Return each passage with its source, version or date where available, and enough surrounding text to interpret an exception.

The wall: retrieval can omit the decisive clause or return a plausible but outdated match. Test whether the necessary evidence is present before measuring answer quality. Give the agent a way to widen its search or read the complete source when an excerpt is inconclusive.

A smaller result set is useful only when it preserves what the decision needs. “Retrieve instead of stuffing” is a selection policy, not an instruction to hide inconvenient context.

Give Subagents Bounded Tasks and Clean Contexts

Use a separate context for exploration whose raw results the main agent does not need. Anthropic's Claude Code subagent documentation describes workers that run in their own context windows and return results to the main conversation.

A worker investigating a log can receive the specific question, relevant service details and a pointer to the log. Ask it to return the finding, supporting location, uncertainty and any unresolved dependency. The main agent can use that result without carrying the entire search trail.

The wall: isolation also removes shared knowledge. Include the constraints the worker needs, and require evidence with its conclusion. Check what your orchestration framework actually passes to a worker; copying the entire parent history would defeat the intended separation. Pasting every worker transcript back into the parent recreates the clutter there.

Use this pattern for separable investigations. A tightly coupled decision that depends on a shared record may be easier to handle in the main context with targeted retrieval.

Order Documents and the Current Ask Deliberately

Make the current task easy to distinguish from its supporting material. Anthropic's long-context prompting guidance recommends placing long documents near the top of the prompt, distinguishing their sources, and placing the query after them. It also recommends locating relevant quotations before answering document questions.

For your application, keep governing instructions in their intended instruction channel. Within the task input, label the current incident record separately from historical notes, then state the question and required output clearly. Do not turn retrieved text into governing instructions merely because you moved it.

The wall: rearrangement cannot restore evidence that was never loaded, and no ordering is a guarantee across every model and task. Compare layouts on your own examples. Avoid repeating an entire rulebook at every turn when a concise, current statement of the task would do.

Make Tool Outputs Selective and Recoverable

Return the result needed for the next decision, with a route back to the full artifact. Anthropic's tool-design guidance recommends meaningful responses and controls such as filtering, pagination and truncation. Its context-editing documentation separately describes clearing older tool results from conversation history.

For a failed check, return the failure, the relevant error excerpt and the saved report's location. For a search, return relevant passages and their sources. For a successful operation, return the status and the identifiers needed to continue. Avoid embedding the whole report simply because the tool produced it.

The wall: blind truncation can remove the decisive error or qualification. Make omissions visible and let the agent request more. Preserve the full output outside active context when it will be needed for inspection.

These choices belong in the request assembly layer. The broader context-engineering guide explains how instructions, memory, tools and retrieved material fit together.

What's Overhyped About Context Rot?

There is no universal message count or window percentage that tells every agent when to reset. A threshold that prevents an oversized request is a capacity setting. The point where a particular workflow becomes unreliable needs to be measured against its actual tasks.

Nor does every bad answer in a long chat demonstrate context rot. Missing evidence, contradictory instructions, a broken tool or an unsupported task can produce similar symptoms. A useful incident report names the failed decision and records what the model received.

Compaction is not lossless, and external memory is not automatic relevance. Treat both as components whose outputs need checks. Retaining the wrong conclusion more efficiently can make a workflow consistently wrong.

Finally, caching is a different intervention. A KV cache reuses attention computation; it does not decide which document matters or remove stale history. A well-cached request can still contain the wrong material for the next decision.

A Short Checklist for This Week

Apply this to an agent with a recurring failure:

• Capture the actual input at the failed decision and identify the evidence it needed.
• Replay the task with focused evidence and unchanged requirements; compare correctness across representative cases.
• Replace the noisiest tool response with a relevant excerpt and a pointer to the full artifact.
• Define a compaction handoff that preserves decisions, rejected approaches, constraints, sources and unfinished work.
• Isolate a separable investigation in a clean worker context, if exploration is the source of clutter.
• Ship the change only if the resumed workflow keeps the necessary facts and improves the targeted failure; keep the prior behavior available to restore.

Start with the failure you can reproduce. That gives the change a clear acceptance test instead of making “less context” the success metric.

Questions About Context Rot

Is context rot still a thing?

Anthropic's current compaction docs still explicitly describe response-quality degradation as conversations grow. That supports continuing to manage context; it does not establish how every current model performs. Evaluate the model and workflow you deploy.

How to avoid context rot?

Keep current instructions and necessary evidence, remove obsolete or redundant material, compact history carefully, and retrieve details when needed. Isolate noisy side tasks where useful. Verify that each change preserves the facts required for the next action.

What causes context rot?

Context length, the relevance of surrounding material and how information is presented can affect reliability. The original study does not establish a single underlying mechanism. For an agent, inspect the supplied evidence and working state before selecting a fix.

Subscribe to the newsletter for practical guides to building and operating AI workflows.