Context Engineering Is Deciding What Earns a Place in the Window

A million tokens. That was supposed to end the argument.

When the big windows landed, plenty of teams quietly deleted their retrieval layer. Chunking, ranking, reranking, all of it felt like work you did because the window was small. Context engineering, the discipline of deciding what actually earns a place in that window, looked like scaffolding that scale would knock down. It did not work out that way.

The window grew. Attention did not improve in proportion, cost still scales linearly with every token you paste, and everything you add competes with everything else for a finite amount of the model's attention. The window is a budget. Most teams are still treating it as a bucket.

What the lost-in-the-middle paper actually measured

The paper everyone cites is Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni and Liang, published in Transactions of the ACL volume 12, pages 157 to 173. Two tasks: multi-document question answering and key-value retrieval. One finding: accuracy is highest when the relevant information sits at the beginning or the end of the input, and it degrades significantly when the model has to reach into the middle. A U-shaped curve, and it showed up even in models explicitly built for long contexts.

For two years that result circulated as folklore and got treated as a quirk of one model generation. It is not a quirk. In 2025, Wu, Wang, Jegelka and Jadbabaie gave it an architecture-level explanation at ICML: causal masking inherently biases attention toward earlier positions, because tokens in deeper layers attend to increasingly contextualised representations of earlier tokens, while distance decay in positional encodings pulls the other way toward recency. The two effects fight each other. The middle is where neither one wins.

That matters because it tells you the shape is structural. You do not out-train a curve that falls out of the mask.

Throwing things away was not a compromise made for cost. It was the accuracy fix.

Context rot, measured across eighteen models

Chroma published the most useful empirical version in July 2025. Kelly Hong, Anton Troynikov and Jeff Huber ran 18 models spanning Anthropic, OpenAI, Google and Alibaba. Every one degraded at every input-length increment tested, including on tasks a first-year student would find trivial. On LongMemEval the gap between a focused prompt of roughly 300 tokens and the full prompt of roughly 113,000 tokens was wide enough to change the answer. The strangest result: shuffled haystacks scored better than coherent ones. Structure in the surrounding text made retrieval harder, not easier.

NoLiMa attacked the same problem from the other side. Strip the lexical overlap between the question and the needle so the model has to infer the association rather than pattern-match it. Twelve models that all claim 128K or more were tested. At 32K, ten of them fell below half their short-context baseline. Your effective window, on anything requiring inference rather than string matching, is roughly an order of magnitude smaller than the number on the pricing page.

2026 kept going. LOCA-bench, from Zeng, Huang and He in February, grows an agent's context arbitrarily while holding task semantics fixed, which finally separates the task got harder from the context got longer. And When Attention Closes, from Dongre and colleagues in May, proposed a Goal Accessibility Ratio measuring how much attention generated tokens still pay to the tokens that defined the task. A causal ablation in Mistral collapsed recall on a 20-fact task to 11 percent. The goal was still sitting in the residual stream. The model just could not reach it.

Retrieval versus stuffing, with the actual numbers

Here is the result that should make retrieval advocates uncomfortable. In June 2026, Hamilton and colleagues published a direct comparison on document-grounded question answering. Long-context prompting scored 73.1 percent correctness. Semantic RAG scored 65.4 percent. Stuffing won.

It won at 26 times the per-query token cost. They call it the token tax, and 7.7 accuracy points for 26x spend is a trade some workloads should take and most should not. The honest reading is neither that retrieval beats stuffing nor the reverse. Almost nobody has measured their own version of that number, and the default gets chosen by whichever pattern the team already had lying around.

The other thing raw accuracy hides: a 1M label is a capacity, not a guarantee. On MRCR v2 with eight needles at the full million tokens, the spread between models carrying identical 1M badges has run past fifty points. Claude Opus 4.6 was reported at 76 percent while Gemini 3 Pro sat near 25 percent on third-party aggregation, and both were sold as million-token models. Same label, different machine.

Compaction is the discipline of throwing things away

The cleanest 2026 result on this comes from Lodha, Pahlavikhah Varnosfaderani, Chakraborty and Mithal, June 2026, on a 50-task enterprise expense-itemisation benchmark. Full conversation history: 71.0 percent completion. Pruning to the last five tool calls: 79.0 percent. Pruning plus summarisation: 91.6 percent. Same model, same tasks, less context.

The cost side is the part people skip. Full history burned 1,480,996 tokens and 14.56 hours per benchmark run. The pruning approach used 535,274 tokens and 5.79 hours. Roughly a third of the tokens, twenty points better, two and a half times faster. Throwing things away was not a compromise made for cost. It was the accuracy fix.

Providers have started shipping this as infrastructure rather than advice. Anthropic's context editing beta clears the oldest tool results once input crosses a default trigger of 100,000 tokens, keeps the three most recent tool uses and lets you set a clear_at_least floor so you never invalidate a cache to save four thousand tokens. Moonshot published its own version in the Kimi K2.6 notes: a discard-all strategy where, past the threshold, only the most recent round of tool messages survives, inside a 262,144-token window. Blunt, and it works.

Tool results need a stable shape and a fixed address

Most context bloat is not documents. It is tool output. A browser fetch, a database dump, a log tail, each serialised however the library felt like serialising it that day.

Two rules pay for themselves. First, fix the schema and keep it fixed: same key order, same field names, no wall-clock timestamps in the envelope, no per-call UUIDs. Second, give evidence stable addresses. Number your retrieved blocks and keep the numbering deterministic so the model can refer to block 7 rather than re-quoting four hundred tokens of block 7 back at you. Citation by index is cheaper than citation by copy, and it makes downstream checking mechanical instead of fuzzy.

In automation and robotics work the pattern is familiar from the factory floor: the fixture that holds the part identically every cycle is worth more than the clever gripper that holds it differently each time. Determinism upstream buys you everything downstream. In context terms, what it buys you is the cache.

Prompt caching turns context order into an economic decision

Caching is a prefix match, and that single fact reorganises how you lay out a prompt. Anthropic charges 1.25x base input for a 5-minute cache write, 2x for a 1-hour write and 0.1x for a hit, which means the 5-minute cache pays for itself after one read and the 1-hour cache after two. OpenAI's guide puts GPT-5.6 and later on the same footing: a 1,024 visible-token minimum, 1.25x on the write, 0.1x on the read, with a 30-minute floor after the most recent write or reuse.

Because it is a prefix match, one changed byte anywhere in the prefix recomputes everything after it. So the ordering rule stops being editorial and becomes arithmetic: frozen system prompt first, deterministic tool list second, retrieved evidence third, volatile history and the live turn last. A datetime.now() in your system prompt is not a cosmetic bug. It is a bill.

DeepSeek prices this more aggressively than anyone. On V4-Flash a cache hit costs $0.007 per million tokens off-peak against $0.22 for a miss, about 31 times cheaper, inside a 1M window with 384K maximum output. And their pricing table has a tell you only notice if you live here. Peak hours are 01:00 to 04:00 and 06:00 to 10:00 UTC, Monday to Friday. From my desk in Shenzhen that reads as 09:00 to 12:00 and 14:00 to 18:00 local: a Chinese working day, with the two-hour lunch break cut neatly out of the middle. Off-peak is half price. That schedule is not an abstract load curve, it is an office calendar, and a team batching from Lagos or Berlin is collecting the discount by accident.

One more budget trap. Anthropic's own documentation notes that Claude 4.7 and later use a newer tokenizer producing roughly 30 percent more tokens for the same text. Your context budget is not denominated in characters, and it shifts under you when you change model. Meanwhile the full 1M window now bills at standard rates, so a 900K request costs the same per token as a 9K one. The price stopped punishing long context. The attention did not.

Measuring context quality instead of guessing

Four measurements, none of them expensive.

Position sweep. Move the answer through the window at fixed intervals and plot accuracy. This is literally the 2023 experiment and it takes an afternoon. If your curve sags, you now know where not to put evidence.

Distractor injection. Add plausible near-misses of the kind a keyword search would surface. Chroma showed that a single distractor hurts and several compound, which means a retriever tuned purely for recall is actively degrading your context.

Block ablation. Drop each context block and rerun. If the score does not move, that block was decoration and you have been paying for it on every single turn. This is the test that most directly answers what earns a place.

Cache hit rate as a product metric. If cache_read_input_tokens comes back zero across repeated requests with a supposedly identical prefix, something is moving: a timestamp, an unsorted dictionary, a tool list assembled from a Python set. Put it in the trace viewer next to latency and cost, not in a one-off script somebody runs in a panic.

The strongest objection: this is temporary scaffolding

The objection is correct, and it is stronger than most people making it realise. Models are getting genuinely better at long context, fast. On the MRCR v2 eight-needle leaderboard for contexts up to a million tokens, mid-2026 entries include Gemini 3.7 Flash at 0.970, Qwen3.8 Max at 0.929 and GPT-5.6 Sol at 0.915. Claude Sonnet 4.5, roughly eighteen months earlier, scored 18.5 percent on the same test at 1M. That is not incremental. If you built an elaborate chunking pipeline in early 2025 to work around a 32K effective window, most of that code is dead weight now, and a fair amount of the context hygiene advice written in 2024 has aged into superstition.

Three things survive anyway. Cost does not degrade, it scales. A 900K-token prompt at $5 per million input tokens is $4.50 before the model emits a character, and an agent that takes 200 turns pays some version of that 200 times. Better attention does not make arithmetic cheaper.

Second, agents fill windows faster than vendors grow them. Windows went from 128K to 1M in about two years. A single tool-using turn can emit 50,000 tokens of browser output. That expense benchmark burned 1.48 million tokens on fifty tasks. Supply moves on a slow linear curve set by research labs. Demand is set by whatever your tools happen to print.

Third, the measurement outlives the workaround. Position sweeps, distractor tests and ablations are not compensating for a model weakness. They are how you find out which parts of your context are doing work. That question survives every capability jump, and it gets more expensive to answer by intuition as the window gets larger.

The two curves are moving in opposite directions and nobody publishes the crossing point. In the same year that the best models retrieved above 0.90 across a million tokens, a fifty-task expense workflow needed 1.48 million tokens to reach 71 percent, then reached 91.6 percent on roughly a third of them. Somewhere between those two numbers is the version of this piece that stops being worth writing. I have not seen the eval that tells me when we get there.

Tools referenced

vLLM, reviewed here: vLLM review.

LangGraph, reviewed here: LangGraph review.

LlamaIndex, reviewed here: LlamaIndex review.

RAGFlow, reviewed here: RAGFlow review.

pgvector, reviewed here: pgvector review.

Qdrant, reviewed here: Qdrant review.

Sources

Liu et al., Lost in the Middle: How Language Models Use Long Contexts, TACL vol. 12 (2024): https://aclanthology.org/2024.tacl-1.9/

Chroma Research, Context Rot: How Increasing Input Tokens Impacts LLM Performance (14 July 2025): https://www.trychroma.com/research/context-rot

NoLiMa: Long-Context Evaluation Beyond Literal Matching (arXiv 2502.05167, ICML 2025): https://arxiv.org/abs/2502.05167

Wu, Wang, Jegelka and Jadbabaie, On the Emergence of Position Bias in Transformers (ICML 2025): https://arxiv.org/abs/2502.01951

Lodha et al., Less Context, Better Agents: Efficient Context Engineering for Long-Horizon Tool-Using LLM Agents (June 2026): https://arxiv.org/abs/2606.10209

Hamilton et al., The Token Tax of Epistemic Accuracy: Comparing RAG and Long-Context Architectures (June 2026): https://arxiv.org/abs/2606.20898

Anthropic, Pricing documentation (caching multipliers, long-context pricing, tokenizer note): https://platform.claude.com/docs/en/about-claude/pricing

DeepSeek API, Models and Pricing (cache hit and miss rates, peak and off-peak windows): https://api-docs.deepseek.com/quick_start/pricing