AI on a Long Document: Don't Just Stuff the Context

When you need AI to work over one long document, a 90-page contract, a quarterly report, a commercial lease, the method that works is set by what you need out of the document, not by how many pages it has. If you need a few specific fields, locate the right section first and read only that. If you need a faithful summary, split the document, summarize the parts, then summarize the summaries. Stuff the entire thing into a single call only when the task needs reasoning across the whole document that you cannot localize. The reflex to paste the whole file into a million-token model and ask a question is the expensive, less accurate default, because a document that fits the context window is not the same as a document the model actually read.

That last point is the one most tutorials skip. They open by celebrating the size of the context window, as if fitting the file were the whole problem. Fitting it is the easy part. Getting the model to use all of it reliably, on every run, without paying for the full document each time, is the part that decides whether the automation holds up.

Three ways to run a long document through a model

There are really only three that matter, plus one anti-pattern people reach for by habit.

Stuff. Put the whole document in the prompt and ask. Simplest to build. One call, one place to look when something is wrong. The cost is that you pay for every token on every run, and the model has to attend to material spread across the entire file.

Map-reduce. Split the document into parts, run the model on each part independently to produce an intermediate result, then combine those results in a final call. This is the standard way to summarize something larger than the context window, and it scales because no single call holds the whole file. Google Cloud, LangChain, and most production summarization stacks all describe the same shape.

Locate-then-extract. Read only the part of the document that holds the answer. Route each field to its section using the document's own structure, a table of contents, headings, page ranges, or a cheap first pass that finds where the answer lives, then run a small extraction call against just that section. This is the right move when you need specific values out of a structured document and do not need to reason over the rest of it.

The anti-pattern is embedding a single document into a vector database and retrieving chunks from it. That is a tool for a corpus you query repeatedly, not for one file you already hold in full. We covered where retrieval actually earns its keep in do you need a vector database for AI on your docs. For a single long document, embedding it adds a store, an embedding job, and chunk boundaries that can split a rule from its exception, and buys you nothing a direct read would not.

"It fits the context window" is not "the model read it"

This is the correction the size-first advice misses. Anthropic's own documentation states it plainly: "As token count grows, accuracy and recall degrade, a phenomenon known as context rot. This makes curating what's in context just as important as how much space is available." More context is not automatically better.

The effect has a name and a measured shape. The Lost in the Middle study, published in the Transactions of the Association for Computational Linguistics, found that model accuracy follows a U-shaped curve across the position of the needed information: highest when the relevant fact sits at the start or end of the context, and degrading by more than 30% when it sits in the middle. The finding replicated across six model families and has held up since.

It has also gotten worse to measure as claimed windows grew. The NoLiMa benchmark from Adobe Research, published in 2025, tested whether models can find information that requires a small inference rather than a literal keyword match. Most models it tested fell to half their short-context accuracy or worse by 32,000 tokens, and even a top performer like GPT-4o dropped from 99.3% at short lengths to 69.7% as the context grew. The practical reading: a model's effective usable length is often far shorter than the window it advertises.

None of this means stuffing is broken. For a knowledge base where the relevant passage is short and you place it well, reading the whole thing avoids the retrieval misses that come with chunking. But when the task is to extract from or summarize across a whole long document, the model has to attend to material scattered throughout the file, and that is exactly where context rot bites. So if you do stuff, put the instruction and any anchor near the top of the prompt, restate the question at the end, and do not assume a clause on page 44 of a stuffed 90-page prompt gets read as reliably as one on page 2.

Pick the method by the shape of the task, not the size of the file

Here is the decision we actually use. Read it by what you need out of the document, because that, not the page count, is what should choose the method.

What you need from the document	Best method	Why
A handful of specific fields (dates, amounts, named clauses)	Locate-then-extract	Reads each field from a short, focused context where recall is high; pays for a fraction of the tokens
A faithful summary of the whole document	Map-reduce	Scales past the token limit; each part is summarized on its own before combining
A judgment that depends on the whole document at once (does clause A contradict clause C on page 60)	Stuff, if it fits	The reasoning cannot be localized to one section, so the model needs it all in view
The document is larger than the context window	Map-reduce, or locate-then-extract	It cannot fit no matter how cheap tokens get, so it must be processed in parts
One document, but you will ask many different questions of it over time	Reconsider as a corpus	At that point retrieval starts to earn its keep; see the vector database decision

Most automation work lands in the first two rows. Someone wants six values out of every incoming contract, or a two-paragraph summary of every report. Neither needs the whole document held in one call, and building it that way is how a workflow that felt cheap in testing turns into a bill nobody can explain.

A worked example: six clauses from a 90-page contract

Say the job is to pull six fields out of every incoming vendor contract: governing law, termination notice period, liability cap, auto-renewal terms, the assignment clause, and payment terms. A 90-page contract runs about 45,000 words, roughly 60,000 tokens.

The stuff approach sends all 60,000 tokens on every run and asks for the six fields. Two problems. First, cost: you pay for the full contract each time, and if you process a few hundred a month that adds up fast for no reason. Second, accuracy: those six clauses are scattered through the document, and several sit in the middle sections where recall is weakest. The liability cap on page 51 of a stuffed prompt is precisely the kind of fact context rot swallows.

Locate-then-extract handles the same job differently. A contract has structure: numbered sections, a defined-terms block, often a table of contents. Use that structure, or a cheap first pass over section headings, to route each field to its one to three pages. Then run a small extraction call, a few hundred to a couple of thousand tokens, against just that section, and ask for structured output so the result parses cleanly. Total tokens drop by roughly an order of magnitude, and every field is read from a short context where the model attends well.

Then verify, because a clean-looking extraction is not a correct one. A misread of 1,240.00 as 1,240.60 parses perfectly and is still wrong. The fix is a deterministic check the document hands you for free, which is the whole argument in how to verify AI document extraction: confirm a date parses, a cap is a number in a sane range, a notice period is a real duration. Extraction pulls the value; the check decides whether to trust it.

The tempting fourth option, embedding the contract into a vector store and retrieving clauses, adds a store, an embedding step, and chunk boundaries that can separate the auto-renewal clause from the carve-out that limits it. For a document you already have open in front of you, that is retrieval risk with no retrieval upside.

The failure modes

These are the five ways long-document processing actually goes wrong, in the order I see them.

1. Middle-of-document miss. You stuffed the whole file, and the one fact you needed sat in the low-recall middle. It fit, so nobody suspected the model never really used it. This is context rot, and it is invisible unless you check specific fields against ground truth.
2. Reduce-step drift. In map-reduce, the per-part summaries are usually fine. The combine step is where a summary distorts, because that call compresses many inputs at once and can drop or overstate. Keep the reduce input short and structured, and if the document is very long, merge summaries in pairs through a couple of layers rather than all at once.
3. Chunks that split the unit of meaning. Splitting on a fixed token count lands a rule in one chunk and its exception in the next, and the model answers from half the picture. For documents that have structure, chunk on that structure, sections, pages, headings, not on an arbitrary token boundary.
4. Embedding a document you already have. Reaching for a vector store on a single file imports every retrieval failure mode, stale index, missed synonym, split answer, to solve a problem a direct read does not have.
5. Paying full tokens every run. Stuffing bills the entire document on each execution. If the task only needs a few fields, locate-then-extract does the same job for a fraction of the cost, and the saving compounds with volume.

How to decide this week

Answer three questions before you write any code. First, what do you actually need out of the document: a few fields, a faithful summary, or a judgment that depends on the whole thing at once? That answer picks the method more than anything else. Second, does the document have structure you can route on, headings, sections, a table of contents, predictable page ranges? If it does, locate-then-extract is usually the cheapest accurate build. Third, is this one document or a growing set you will query repeatedly over time? If it is a set, you are drifting toward a corpus, and the vector database decision applies instead.

For field pulls, start with locate-then-extract and verify each value with a deterministic check. For summaries, use map-reduce and keep the reduce step small. Reserve full-context stuffing for the genuine case where the reasoning cannot be localized, and when you do stuff, place the instruction and the key material where the model attends best. Designing that boundary correctly, and not defaulting to the most impressive-sounding option, is most of what we do when we build custom AI software for clients. If you want a second opinion on which method your document workflow should use, tell us what you are processing and we will give you the simplest build that holds up in production.