Stale RAG Data: When AI Answers From Deleted Docs

A RAG system keeps citing documents you already edited or deleted because the pipeline that built it only knows how to add. It embeds new content into the vector store and never removes the old, so an edited or deleted source leaves behind an orphan vector that still matches queries. That orphan scores just as high as a fresh chunk, the model retrieves it, and your bot confidently answers from a policy you rewrote last month or a price you dropped last week. The fix is not a better embedding model. It is to treat the index as a mirror of your system of record: key every chunk to its source document, delete by that key before you re-upsert on any change, and run a reconciliation sweep to catch what the update events miss.

Building the index is the demo. Keeping it in sync with data that changes every day is the actual product, and it is the part almost every RAG tutorial skips.

Why your bot answers from data you already changed

Similarity search has no sense of time and no sense of truth. When a query comes in, the vector store returns the chunks whose meaning is closest to the question. A stale chunk about your old refund policy is semantically identical to the current one. It matches just as strongly, comes back with just as high a score, and there is nothing in a raw vector that says "this is out of date."

So the failure is silent. Nothing errors. No run turns red. The bot returns a fluent, plausible answer sourced from a document that no longer reflects reality. The damage is distributional: across hundreds of questions, accuracy drifts down as more of the index goes stale, and no single log line tells you it happened. This is worse than a crash, because a crash gets noticed.

Most pipelines get here honestly. The build-once tutorial loads a folder, chunks it, embeds it, and ships. That is correct for the launch. It is wrong for week two, when someone updates the handbook, archives a product page, or deletes a customer record, and the index does not hear about any of it.

The three change events, and what breaks in each

Every source document does one of three things over its life. A build-once pipeline handles exactly one of them.

Change event	What a build-once pipeline does	What breaks	The fix
Document added	Embeds and inserts it	Nothing. This is the only case tutorials cover	Keep doing this
Document edited	Often re-embeds and inserts the new version alongside the old	Two versions now match the query. The model may retrieve either, and answers flip between old and new	Delete all chunks for that source, then re-chunk and insert the new version
Document deleted	Nothing	Orphan vector stays and gets cited as current, forever	Delete all chunks for that source on the delete event, plus a sweep to catch misses

The edit case is the sneaky one. If your update step upserts the new chunks without first deleting the old ones, you do not overwrite the document, you duplicate it. Now the index holds two answers to the same question and picks between them by cosine distance. That is how a bot gives one answer on Monday and the opposite on Thursday with no code change in between.

The build: key every chunk, delete before you upsert

The whole fix rests on one decision you make before the first embedding: give every chunk a deterministic ID derived from its source document, not a random UUID.

Build the ID from the source, like handbook-hr#0, handbook-hr#1, handbook-hr#2. A hierarchical, prefixed scheme means you can find and remove every chunk of a document later in a single operation. In Pinecone, upsert with the same ID overwrites that vector's values and metadata, and you delete by ID; serverless indexes support delete by ID prefix, which is exactly why the prefix convention matters (as of 2026, serverless indexes do not support delete by metadata filter). In pgvector, because it is just Postgres, removal is a plain DELETE FROM chunks WHERE source_id = 'handbook-hr'. Either way, a stable ID is what turns "remove this document" from impossible into one call. If your IDs are random, you have no handle to grab.

Then add a content hash. Store a hash of each source document's text alongside it. When a change event fires, compare the new hash to the stored one. If it matches, stop, because the content did not really change and re-embedding it is pure waste. If it differs, run the update as a delete-then-insert, never an in-place patch:

1. Delete every chunk for that source ID.
2. Re-chunk the new version of the document.
3. Embed and insert the new chunks.
4. Store the new content hash.

Delete-then-insert, not update-in-place, because editing a document shifts its chunk boundaries. Chunk three of the old version is not chunk three of the new one, and if the edit made the document shorter, a per-chunk overwrite leaves the old tail chunks stranded as orphans. Wipe the source's chunks as a set and rebuild them as a set. It is simpler to reason about and it has no orphan failure mode.

Whether the change event arrives as a webhook from your CMS or Drive, or as a scheduled diff that compares hashes, is the same real-time-versus-batch call you make for any trigger. Fast-moving, high-stakes corpus: fire on the change. Slower corpus: a nightly hash diff is less to operate. We walk through when RAG is even the right architecture in do you need a vector database for AI on your docs; this article is the other half, how to run one once you have committed to it.

"Delete" is not always delete

There is a second, sharper trap hiding inside the delete step, and it matters most for exactly the data you are most obligated to remove.

Many vector databases implement deletion as a soft delete. They mark the vector as removed at the metadata level and skip it at query time, but leave the actual numbers in the index to avoid an expensive rebuild. That is fine for a stale policy. It is a problem when the deletion is a legal obligation. A June 2026 paper titled Ghost Vectors showed that soft-deleted embeddings in HNSW indexes remain reconstructible across Chroma, FAISS, Weaviate, and Pinecone, meaning the original vector can be recovered after it was supposedly deleted.

So if a customer asks you to erase their data, or a document falls under a retention limit, a delete flag is not erasure. The vector is still in memory, still reconstructible, still a copy of information you told someone you removed. Real erasure needs a hard compaction or an index rebuild that evicts the vector, not a tombstone that hides it. This is the same shadow-copy problem we flag for execution logs in PII in automation logs: the record you think you deleted often lives on in a store you forgot you were keeping. Your vector index is one of those stores.

A reconciliation sweep catches what events miss

Even a correct update pipeline drops events. A webhook fails to fire, a re-embed run errors out and nobody notices, someone edits a row directly in the database and skips the app entirely. Every one of those leaves the index a little out of sync, and event handlers alone never fully converge.

So put a sweep on top, on a schedule. Pull the list of live source IDs from your system of record. Pull the distinct source IDs present in the index. Then reconcile the two: delete index entries whose source is gone, and re-embed sources whose stored content hash no longer matches. This is the same instinct as inverted monitoring in get alerted when an automation stops running. You are not waiting for a failure to announce itself, you are checking for the absence of something that should be there. As a cheap backstop, stamp each chunk with a source_updated_at in its metadata so you can filter or down-rank anything older than a threshold at query time, which limits the blast radius of a missed update while the sweep is what actually fixes it.

The reason this deserves a first-class step is that a vector store's entire value is being a faithful mirror of the source of record. When we build a system like ma-permits.geninfos.com, the whole point is that its data tracks the authoritative record, because citing a permit that was revoked is worse than citing nothing. A RAG index earns trust the same way, and it only stays trustworthy if something keeps it honest.

What to do next

Before you ship a RAG bot, answer one question in plain words: what happens when a source document is edited or deleted? If you cannot name the exact step that removes the old vector, you already have an orphan problem, it just has not embarrassed you in front of a customer yet.

Do the cheap thing now. Give every chunk a source-derived ID and store a content hash per document before the first embedding goes in. Retrofitting stable IDs onto an index that is already in production usually means re-embedding the whole corpus, so this is the rare case where five minutes of naming discipline up front saves a full rebuild later. If you are standing up an AI assistant on your own documents and want the sync layer built correctly the first time, that is the kind of work we do at custom software platforms; tell us what the corpus looks like and we will tell you whether it even needs a vector store.