Big context, little content – that is the short summary of what research tells us about large LLM context windows. Vendors advertise one million tokens as if capacity were the same as capability. Load everything in, the message goes, and the model will sort it out. Data professionals have heard this promise before. Fifteen years ago the container was called a data lake, and the currency was terabytes instead of tokens. The result was the same: the bigger the container, the easier it became to confuse storing information with using it.
This article looks at what the research says about large context windows, and what it means for data work – with concrete recommendations for daily practice.
A familiar story
Storage became cheap, so many teams stopped asking what belonged in the lake and what did not. Schema-on-read promised that structure could wait until later. Later rarely came. Many lakes turned into swamps: technically complete, practically useless, because nobody could say which of the seventeen copies of the customer table was the right one.
The million-token context window repeats this pattern. It removes a visible limit, and with it the discipline that the limit used to enforce. Capacity does not create clarity. It only delays the moment when the missing clarity becomes visible – and makes that moment more expensive. There are two differences, and both make the context window worse than the data lake.
The first difference is economic. Dumping into the lake was at least cheap – storage cost next to nothing, which is why the habit spread in the first place. Context is billed by the token, on every single call, and large inputs also slow every answer down. But the price per token, the number everyone talks about, is the wrong unit to watch. What counts is the cost per use case: how many tokens it takes – across all calls, retries, and corrections – to complete a task correctly. A bloated context drives that number up twice: every call gets more expensive, and degraded answers force more calls. Paying more to get less is a hard sell in any architecture review.
The second difference is silence. A swamp announces itself through failing pipelines and frustrated analysts. An overloaded context window announces nothing. The model keeps answering, fluently and confidently, while the quality of its answers goes down. And the data practitioner stays in flow, confirmed by the model at every step – until it turns out that the last hour was spent driving full speed into a wall.
What the research shows
Three findings are worth knowing, and they point in the same direction. For readers in a hurry, here is the overview – the details follow below.
| Study | Observation | Consequence for data work |
|---|---|---|
| Lost in the Middle (Liu et al., 2024) | Models use the start and the end of the context reliably; the middle drops off. | Critical rules and schemas belong at the edges of the prompt, not in the middle. |
| Context Rot (Chroma, 2025) | Similar-but-wrong content (distractors) damages reliability – and the model fails silently. | Filter old versions and near-duplicates out of the context – better yet, update or delete them at the source. |
| NoLiMa (Modarressi et al., 2025) | When the model must understand instead of match keywords, the usable window shrinks drastically. | The harder the semantic task, the smaller the input should be – split large tasks instead of filling the window: divide and conquer. |
The oldest and best replicated is Lost in the Middle (Liu et al., TACL 2024). Model performance depends heavily on where in the context the relevant information sits. Information at the beginning and at the end is used reliably. Information in the middle drops off, and the effect grows with context length. Since 2025 the mechanism is understood: the attention architecture structurally favors early tokens, because every later token can attend to them, while a token in the middle is only visible to what comes after it. Newer models flatten the curve through training, but they do not remove it. As a picture, think of a bathtub: solid at both rims, a dip in the middle – and the wider the tub, the longer the dip. For prompt design this means: where information sits decides whether the model uses it.
The second finding comes from Chroma’s Context Rot report (2025), which tested 18 state-of-the-art models. None of them uses its context uniformly; reliability declines as input grows, even on very simple tasks. Two details matter for data work. First, distractors – content that is similar to what you look for but wrong – hurt far more than neutral filler. Second, models fail differently: some refuse to answer when they get lost, others confidently hallucinate. The second behavior is the dangerous one in a pipeline, because nothing downstream will flag it. For data quality this means: near-duplicates and old versions in the context actively damage answers, and nobody gets a warning.
The third finding is the most relevant one for retrieval on enterprise data. The NoLiMa benchmark (2025) tested what happens when question and answer do not share literal keywords, so the model must reason semantically instead of matching patterns. The result: in this benchmark, models with claimed windows of 128K tokens or more often showed effective lengths of no more than about two thousand tokens, and ten of twelve models dropped to half their baseline accuracy at 32K. The exact numbers are specific to the test setup – real tasks will differ – but the direction carries over. Read that again through the lens of a data engineer. As soon as the task looks like a semantic join instead of an exact-match lookup – which is what real questions against real documents look like – the usable window shrinks far below the number on the datasheet. For sizing this means: as soon as the model must understand instead of match, the usable window is a fraction of the advertised one.
These three papers are not isolated results. Long-context behavior has become a research field of its own: benchmarks such as RULER and BABILong report similar gaps between advertised and effective length, and vendors like Anthropic openly describe context as a limited resource that needs engineering. Interestingly, part of the pattern resembles human memory. People also recall the beginning and the end of a list better than the middle – psychologists call this the primacy and recency effect. There is one important difference, though: a human notices when the middle gets blurry. The model does not. The effective context window is a fraction of the advertised one, it shrinks with the semantic difficulty of the task, and the model gives no reliable signal when the line is crossed.
What this means in practice
Most tools already offer a first line of defense: compaction. When the window fills up, the tool summarizes the conversation and frees space. That is useful, and clearly better than nothing. But compaction is a black box. You do not decide what is kept and what is dropped, and you will not notice what was lost – until an answer quietly ignores a decision made earlier in the session. The same logic applies to agentic workflows: an autonomous agent that runs for hours fills its window even faster, and compacts just as blindly. Automatic summarization is a safety net, not a strategy. Whoever wants a reliable context window should stay in the driver’s seat, and that is what the following recommendations are about. One thing they all share: AI can assist with each of them – drafting summaries, flagging duplicates, generating tests – but the judgment about what stays, what goes, and what is still true must remain human. Delegating that judgment back to the model just recreates the black box one level up – critical thinking cannot be outsourced.
The good news: nothing about them is new. It is classical data management, applied to a new container. The recommendations below grew out of my own experience and many exchanges with other data practitioners – each one comes with a concrete example from daily data work.
Curate instead of dump
Everything in the context should be there for a reason; everything else stays outside. The oldest rule of data processing applies here unchanged: garbage in, garbage out – and without active curation, the garbage simply stays in. The Context Rot results sharpen this point: near-duplicates and outdated versions are not just noise, they are distractors that actively pull accuracy down. For many enterprise RAG tasks this means precision matters more than broad recall: five relevant, clean chunks outperform fifty mediocre ones.
Example: An LLM should generate SQL against a data warehouse. Before pasting the schema export into the prompt, ask two questions: Does the model really need all of these tables for this task? And can anything go? Provide only the tables the task actually touches, plus their join keys and a sample row each – and make sure it is the current DDL. If the export still contains CUSTOMER_OLD, CUSTOMER_V2, and CUSTOMER_TMP next to CUSTOMER, the model will happily join against the wrong one. Deleting the obsolete versions from the input does more for correctness than any prompt instruction.
Placement is design
Instructions, constraints, and the decisive facts belong at the beginning or the end of the prompt, never buried in the middle of a long context. The same information performs differently depending on where it sits – prompt layout follows the bathtub curve.
Example: In a prompt that generates transformation logic, state the task and the hard rules first (“ANSI SQL only, no vendor-specific functions, never filter history with a physical DELETE”). The bulk goes in the middle: DDL, mapping specification, sample data. Each rule is stated exactly once, at a rim of the bathtub – a rule buried between line 400 and 500 of the prompt is a rule the model will eventually ignore.
Externalize what must survive
Decisions, definitions, and conventions belong in a persistent place outside the window – and get injected fresh and compact when needed. A long conversation history is a log, not a system of record, and no database rebuilds its state by replaying the full log on every query.
Example: Maintain a CONVENTIONS.md in the repository that the LLM receives at the start of every session: naming standards, the rules of your bitemporal model (valid time vs. transaction time, corrections only through new records, no updates in place), and short architecture decision records – one ADR per decision, with a one-line rationale (“ADR-014: surrogate keys are hash-based, because loads must run in parallel”). And treat this file like code: review changes via pull request, keep one fact per entry, delete entries that no longer hold, and check for duplicates before adding. Clarity, precision, and brevity are the quality criteria – a bloated conventions file just recreates the distractor problem in a new place.
Treat LLM output as data, because it is data
This recommendation follows directly from the research: models give no reliable warning when their context is overloaded, so the only place to catch the damage is the output. An LLM step in a pipeline is an untrusted transformation and needs the same controls as any other – downstream tests are the warning system that the context window itself does not have. Model confidence is not a quality metric. Tests are.
Example: If an LLM generates a dbt model or a mapping, the output passes the same gates as human code: unique and not_null tests on the keys, a relationship test against the referenced dimension, and a reconciliation query that compares row counts and sums between source and target. In review, read the diff, not the summary – the dangerous errors are small, plausible, and syntactically perfect. And when a long session has gone off track, restart with a clean summary instead of patching; the same instinct that says rerun a broken load cleanly instead of fixing it at 2 a.m.
Bottom line
The context window is the newest oversized container in the industry, and it invites the oldest mistake: mistaking capacity for value. The research is clear – models do not use large contexts uniformly, the effective window collapses under semantic load, and no model reliably reports when it happens. The remedy is not exotic: curation over volume, placement as design, persistent conventions over replayed history, and quality gates around every untrusted transformation. And one principle above all: responsibility means making the judgment calls yourself – not delegating them to the model. In my experience, the GenAI era does not need new instincts, only the discipline to apply the old ones – to a container that, once again, is sold by its size and, this time, billed by it too. And if all this sounds like work: it is. GenAI does not remove the effort, it shifts it – the typing has become cheap, the thinking has not.
A large context window is useful infrastructure, not intelligent data management. If you do not curate, place, persist, and verify what goes into it, you are not giving the model more knowledge – you are giving it more chances to be wrong.
