Changed Embedding Models and RAG Search Got Worse

RAG search gets worse after an embedding change because a query vector can only be compared with document vectors made by the same model, at the same dimensions, with the same task setting. If the width changed, the database errors and you find out right away. If it did not, say you moved from OpenAI's text-embedding-ada-002 to text-embedding-3-small, both 1536 numbers wide, every insert and every search succeeds and the bot just starts pulling unrelated chunks. Not every change needs a full re-embed, though. Lowering dimensions on the same OpenAI v3 or Gemini model, or moving between models in Voyage AI's 4 series, can be done without re-embedding a single document.

Most advice on this topic stops at "new model means re-embed into a parallel index and test before cutover." That is correct for a planned upgrade. It does not help the operator whose search got worse last Tuesday and who does not yet know an embedding setting changed at all, which is the more common way this happens.

Why nothing errors when the widths match

A vector database checks shape, not origin. A pgvector vector(1536) column accepts any list of 1536 numbers. It has no idea whether they came from ada-002, text-embedding-3-small, or gemini-embedding-001 asked for 1536 dimensions through output_dimensionality. All three fit. All three are different coordinate systems.

So the search runs. It compares a question embedded in one model's space against documents embedded in another, ranks the results, and returns the top five with scores that look ordinary. Nothing in the response marks them as noise. The language model then answers fluently from whatever it was handed, which is why the first report usually comes from a customer or a staff member saying the bot "got dumber," not from a log.

In automation tools this happens more easily than it sounds. In n8n, the workflow that loads documents into the vector store and the workflow that answers questions each have their own Embeddings sub-node. Someone changes the model on the chat side to save money or try something newer, the insert side keeps the old model, and the two halves of the system quietly stop speaking the same language.

Which changes force a re-embed, and which don't

This is the table we use when someone tells us an embedding setting changed. Facts are from the vendor docs as of October 2026.

Two rows deserve a closer look.

The dimension-lowering row works because these models were trained so the leading numbers carry the most meaning. OpenAI's guide says text-embedding-3 vectors can be shortened by removing numbers from the end without losing their concept-representing properties, provided you normalize them afterwards. Google says the same of its Matryoshka-trained Gemini models and recommends 768, 1536, or 3072. That turns a full re-embed into a SQL update. It also matters for indexing: pgvector's HNSW and IVFFlat indexes support up to 2,000 dimensions on the vector type and 4,000 on halfvec, so a 3072-wide default from text-embedding-3-large or Gemini cannot get an HNSW index as plain vector at all.

The Voyage row is the one real exception to "new model, new index." Voyage designed its 4 series to share one space, so you can embed the corpus once with the expensive model and run every query on the cheap one. That is a legitimate cost lever, and it is specific to that family. Do not assume it holds for anyone else's model lineup.

The query side can drift on its own

Several providers embed queries and documents differently on purpose, and the setting lives in the request, not the model name.

• Cohere requires input_type for Embed v3 and later: search_document when you index, search_query when you search.
• Gemini gemini-embedding-001 takes a task_type such as RETRIEVAL_DOCUMENT and RETRIEVAL_QUERY.
• Gemini gemini-embedding-2 does not support task_type. Google recommends putting the task in the text instead, in the form task: search result | query: {content} for queries and title: {title} | text: {content} for documents, and says the task must be used consistently.
• Voyage prepends a fixed prompt when input_type is query or document, and documents that vectors made with and without it are compatible.

The failure here is a refactor that drops the prefix, a new helper function that forgets input_type, or a migration to gemini-embedding-2 that leaves task_type in the code path where it no longer applies. The corpus is fine. The fix is to make the query request match what the documents were built with, which is only easy if you wrote down what that was.

Normalization changes the ranking, not just the score

OpenAI returns vectors normalized to length 1, which is why pgvector recommends the inner product operator for them: on unit-length vectors it ranks the same as cosine similarity and runs faster. gemini-embedding-2 normalizes truncated sizes automatically. gemini-embedding-001 does not. Google's docs say you must normalize anything other than 3072 yourself.

So picture a team that moves from text-embedding-3-small to gemini-embedding-001 at 768 dimensions and keeps the inner product index they already had. The new vectors have varying lengths, inner product now rewards longer vectors as much as closer meaning, and some documents start winning almost every query. Nothing errors. The cure is one normalization step on both the insert and query paths, or switching the index to cosine distance.

The fix: store a fingerprint and refuse to mix

The re-embed is not the hard part. Knowing you need one is. We put four facts next to every index, or on every row when one table holds more than one:

1. Model, as the exact identifier, for example text-embedding-3-small or gemini-embedding-001.
2. Dimensions, as requested from the API or after truncation.
3. Document-side task setting, such as input_type=search_document, task_type=RETRIEVAL_DOCUMENT, or the prefix template and its version.
4. Normalized, yes or no.

The query path builds its own fingerprint from its own settings and compares before searching. On a mismatch it fails with an error that names both fingerprints. That converts the silent row in the table into a loud one, which is the whole point. It is the same discipline as pinning a model version for generation, applied to retrieval.

When a real model change is due, the fingerprint also tells you what else to reset. Similarity thresholds tuned on the old model mean nothing on the new one. A semantic cache keyed on old query embeddings has to be flushed, or it will keep matching questions in the wrong space. And if retrieval was weak before the change, check whether the right chunk is being found and then outranked before blaming the embedding model, and confirm the index is not full of stale vectors, since swapping models will not fix either.

What to do next

Open the insert workflow and the query workflow side by side and write down the model, dimensions, task setting, and normalization each one uses. If any of the four differ, you have found your problem, and the table above tells you whether it costs a full re-embed or a one-line fix. Then save those four values with the index so the next change cannot happen quietly.

If you run a document assistant or knowledge-base bot and want the retrieval layer built with these checks from the start, that is part of our custom software platforms work, or tell us what your setup looks like.