Emerging concept
Token Shedding
The deliberate removal of low-value tokens, history or context to keep AI systems efficient, focused and within budget.
An emerging concept in AI systems engineering.
- Status
- Emerging
- Field
- Context management · Inference efficiency
- First tracked
- July 2026
- Related
- context windows · context compression · memory management · context rot · inference cost
A note on status: the problem this term describes is real and already being worked on across the industry. The terminology is not settled — this term is one candidate among several possible framings. This site exists to define it precisely and to track whether the term, or the category it names, gains adoption.
§ What it means
Many long-running AI systems accumulate tokens: conversation history, tool outputs, retrieved documents, intermediate reasoning. Most of it loses value quickly. A stack trace matters until the bug is fixed; a search result matters until it has been summarized; small talk from an hour ago may never matter again. But by default, all of it stays in the model's context window, where it costs money on every subsequent call and degrades the model's attention.
Token Shedding is the practice of removing that material deliberately. Rather than truncating blindly when the window fills, a shedding policy decides which tokens have stopped earning their place — and drops, compresses or offloads them while preserving what still matters.
The name borrows from load shedding in power grids: a controlled, selective reduction that keeps the system healthy, as opposed to an uncontrolled failure when capacity runs out. The distinction from simple truncation is the same. Truncation cuts from the oldest end regardless of value. Shedding is value-aware: old but critical instructions survive, while recent but spent tool output goes.
§ Why it matters now
Three trends converge here. First, agents now run for hours or days, not single exchanges — coding agents, research agents, operations agents — and their working histories can grow continuously unless the system actively removes, compresses or offloads material. Second, inference cost scales with context length on every call, so a bloated window is a recurring tax, not a one-time one. Third, research through 2025 showed that performance itself degrades as windows fill: models attend worse in long, cluttered contexts, a phenomenon practitioners began calling context rot.
Platform vendors have started shipping the primitive. Context editing, automatic compaction and memory tools that summarize-then-drop are now built into major agent products and APIs. What does not yet exist is shared vocabulary and shared practice: which tokens to shed, when, how aggressively, and how to verify nothing essential was lost.
Bigger context windows do not dissolve the problem — they raise the ceiling while cost and attention degradation still scale with what you keep. Deciding what to remove is becoming as important as deciding what to retrieve.
§ What it could include
-
Relevance scoring
Estimating which tokens in the window still contribute to the task at hand, and which have gone stale.
-
History compaction
Replacing long stretches of conversation or reasoning with dense summaries that preserve decisions and state.
-
Tool-output pruning
Dropping raw tool results — logs, search results, file dumps — once their useful content has been extracted.
-
Memory offloading
Writing material to external storage before removing it from context, so it can be re-retrieved if needed.
-
Budget enforcement
Hard token and cost ceilings per task or per turn, with shedding as the mechanism that keeps a system under them.
-
Cache-aware shedding
Removing tokens in ways that preserve prompt-cache prefixes, so shedding reduces cost instead of invalidating it.
§ Emerging signals
A running log of research, products and standards work relevant to this concept. Curated by hand; newest first. Each entry separates the factual record from our interpretation.
-
Anthropic ships context editing and a memory tool
Anthropic introduced context editing and a memory tool on the Claude Developer Platform, enabling agents to clear stale tool results and offload information to external files. Anthropic reported a 29% improvement on internal agentic evaluations from context editing alone, and 39% combined with the memory tool.
Our read — Token shedding shipped as a platform primitive — with published numbers suggesting that removal improves performance, not just cost.
Anthropic
-
Chroma publishes the "Context Rot" technical report
Chroma published "Context Rot: How Increasing Input Tokens Impacts LLM Performance", a technical report showing that model performance degrades as input length grows, even on simple tasks.
Our read — Keeping everything has a quality cost, not just a dollar cost. Degradation with length is the core motivation for value-aware removal.
Chroma
-
Anthropic launches prompt caching
Anthropic launched prompt caching in public beta, letting developers reuse cached prompt prefixes at roughly 10% of the base input price, with cache writes at a 25% premium.
Our read — Caching changed the economics of retained context — and made naive mid-context deletion expensive, creating the need for cache-aware shedding policies.
Anthropic
-
LLMLingua demonstrates aggressive prompt compression
Microsoft Research published LLMLingua, demonstrating prompt compression of up to 20x with limited performance loss on downstream tasks.
Our read — Early evidence that most tokens in a real prompt carry little marginal value — the premise token shedding is built on.
Microsoft Research
§ Track the term
Tracking how this concept develops. Get occasional updates when the term — or context-efficiency practice — starts moving.