AI Context Economics: Retrieval Without the Multiplier

2026-08-30 · Engineering ·

You know the ritual. A new agent session opens, and the first twenty minutes are archaeology: the model re-reads files it has already read, re-derives a decision someone made last Tuesday, re-runs a search another session already ran. Every new AI session starts without enough knowledge of previous decisions, and you pay for the amnesia twice — once in tokens, once in your own supervision. The instinct is to fix it with a bigger context window or a retrieval index. The honest question is what either of those is actually worth, in measured terms.

Section 7 of the Koinessa white paper answers that question in a way most vendors will not: with a five-row evidence table, an explicit refusal to multiply the rows together, and a row of the operator's own telemetry labeled as a receipt, not a benchmark. This post walks through what each measurement actually is, what it does not establish, and why the architecture keeps canonical state outside the model window — including a straight answer to Anthropic's own argument against retrieval indexes.

No compound multiplier, on purpose

The white paper opens the section with a sentence you rarely see in AI marketing: there is no observed whole-task or 70-times result. The external studies and internal receipts measure non-equivalent components — a QA benchmark here, a code-graph system there, a token-variance study somewhere else — so they are not multiplied into a headline. Any future internal projection must declare its assumptions and use either a measured sequential factor or a weighted equation, and it remains a conditional design calculation, never a benchmark.

This is worth dwelling on, because the temptation is real. If semantic retrieval is about 57% cheaper per answer and a structural index uses about ten times fewer tokens, a marketer would multiply, add the telemetry, and print a number. The paper declines, and that refusal is the point. An engineering organization that will eventually hand this system cost-attribution data needs the inputs to stay honest more than it needs the headline to be large. The same discipline runs through the series' other measurement posts, from the verification gap analysis onward: a claim you cannot defend in a client meeting is a liability, not an asset.

What the five measurements actually say

Rafiei Oskooei and colleagues published an empirical study of repository-level code question answering in 2026, comparing approaches on read-only Python repositories (arXiv:2608.01507). The measured boundary the paper quotes: $0.32 per correct semantic answer versus $0.74 for the Deep agentic-search approach — semantic was about 57% cheaper, Deep about 2.3 times as expensive. The largest token gap, 22.4 times, applied specifically to Qwen3-235B, not to every model in the study.

Two qualifiers matter. The task was read-only question answering on Python repositories, so the result does not establish whole-task performance on real engineering work. And the 22.4-times figure is a maximum for one model, so it does not establish every-model savings. What it does establish is narrower and still useful: for the class of work agents do most — finding and understanding code — retrieval-first pricing is materially better than let-the-agent-explore pricing.

A persistent code graph versus exploration

Vogel and colleagues built Codebase-Memory, an open-source system that constructs a persistent, Tree-Sitter-based knowledge graph over a repository and serves it through the Model Context Protocol, parsing 66 languages (arXiv:2603.27277). Evaluated across 31 real-world repositories, it achieved 83% answer quality versus 92% for a file-exploration agent — at about ten times fewer tokens and 2.1 times fewer tool calls, with exploration retained as a fallback.

Read the trade plainly: you give up nine points of answer quality and get back roughly an order of magnitude in tokens. For a first hop before an agent starts work, that is a good trade. It does not establish safe edits, and it does not replace the exploration fallback — the white paper says both, and Codebase-Memory's own design agrees by keeping the explorer available.

How unpredictable the bill really is

Bai and colleagues ran the first systematic token-consumption study of agentic coding: trajectories from eight frontier models on OpenHands over SWE-bench Verified (arXiv:2604.22750). The findings the paper quotes are sobering: roughly a 1,000-times cross-run token difference and up to 30-times variance on the same task. The study also found that higher token usage does not translate into higher accuracy, and that frontier models systematically underestimate their own token costs when asked to predict them.

The white paper's qualifier: eight models, one benchmark harness — this does not describe every agent workflow. But if your cost model assumes an agent session is a predictable quantity, this study is the measured counterargument. Variance that large is an argument for managing context deliberately rather than letting it sprawl, because the sprawl is where the variance lives.

Where the input tokens go

Anthropic's own Claude Code documentation carries a worked session-cost example (code.claude.com). The white paper is careful about provenance here: the 85.6% figure — the share of input-side processing that is context carriage rather than new work — is the paper's own derivation from that page's worked example, with the arithmetic preserved in the research digest; the page itself does not print the percentage. Cache reads alone account for about 51%.

The qualifier: this is a worked example, not realized organization spend. Its function is structural — it shows where the money goes inside a session, which is what makes the next row interpretable.

What Koinessa's own telemetry shows

The fifth row is internal: 27 recorded days of live telemetry through 2026-08-16, with no later day ingested as of 2026-08-23. In that window: 22,696,191,284 total tokens, of which 20,918,082,610 were cache reads — 92.17% — list-priced at $12,027.14. "List-priced" means the provider's published rates applied to the metered tokens, not an invoice.

The paper refuses to call this a controlled productivity comparison, because it is not one. The telemetry identifies where cost accumulated; it does not prove how much a counterfactual workflow would have spent. What it does show, consistent with the Anthropic derivation above, is that in a real AI-first operation the overwhelming majority of billed tokens are history being re-carried — the amnesia tax, metered. The controlling KPI the paper names is verified work per dollar, attributed per work item and interpreted with task quality, acceptance, and human-review cost intact. Tokens are the denominator; verified outcomes are the numerator.

Why the model window is not a filing cabinet

The cheapest context strategy would be to just buy a bigger window and keep everything in it. Two bounded results explain why the architecture declines.

The first is about what survives compaction. Anthropic's engineering writing on managed context states the boundary plainly: compacted messages remain in active context only in summarized form, and are recoverable in full only when separately persisted (anthropic.com). The white paper adds its own precision: lossiness is an inference about unpersisted detail, not a claim that every durable log is destroyed. In other words, if a decision only ever lived inside the window, its full form is one compaction away from being a summary.

The second is about what the window can even do with what it holds. NoLiMa — Modarressi and colleagues, ICML 2025 — is a long-context benchmark that extends needle-in-a-haystack testing by removing literal lexical overlap between question and needle, so models must infer latent associations rather than pattern-match (arXiv:2502.05167). Evaluating 13 models claiming at least 128K-token contexts: at 32K tokens, 11 of the 13 dropped below 50% of their own strong short-length baselines. Even GPT-4o, a top-performing exception, fell from an almost-perfect 99.3% baseline to 69.7%. The white paper is careful to call these benchmark-specific results, and Chroma's Context Rot research points the same direction. The bounded conclusion is not "long context is useless" — it is that the window is a working memory with measurable degradation, not an archive. Canonical state belongs outside it, in something queryable.

The Anthropic counterposition, answered

The strongest argument against retrieval-first comes from Anthropic itself, and the paper states it rather than hiding from it: Anthropic tried a retrieval index inside early Claude Code and removed it in favor of agentic search, citing staleness, privacy, and reliability.

Those are architectural objections, so they get architectural answers. Staleness: the Koinessa code-intelligence index is refreshed per commit by the same hooks that keep evidence current — the 2026-08-23 snapshot held 17,903 symbols for this project alone, last indexed the same day. Privacy: the index is local-only and never leaves the machine, which fits the local-first cost model the series describes elsewhere. Reliability: grep remains the declared fallback rather than a banned tool, and the standing rule is machine-enforced the other way — a pre-tool hook refuses identifier lookups by text search and answers with the exact index query to run instead. Retrieval-first is a cheap first hop, not a doctrine that the index replaces search. The architecture breakdown covers how the index, the Brain, and the Context Compiler fit together as components.

The takeaway

Every new session starting from zero is not a prompt-engineering problem; it is a context-economics problem, and it has measured bounds. Retrieval priced at $0.32 versus $0.74 per correct answer in one study, ten-times token reduction with a nine-point quality trade in another, 1,000-times variance warning against unmanaged sprawl, 85.6% of input-side processing derived as context carriage, and 92.17% cache reads in 27 recorded days of live telemetry all point the same way — without any of them licensing a compound headline. The response is persistent shared memory and retrieval before repetition: canonical state persisted outside the model window, a local index as the cheap first hop, exploration as the declared fallback, and verified work per dollar as the only KPI that counts. The full argument, with every qualifier intact, is in the Koinessa white paper; the companion post on governed throughput picks up where the cost floor ends.