Semantic Caching for AI: Cut Cost Without Wrong Answers

Semantic caching stores past AI answers and returns one when a new request is close enough in meaning to a request you already paid for, skipping the model call entirely. It is not the same thing as prompt caching, which only discounts the repeated input prefix of a call you still make. The catch is that "close enough" is a number you set, and set too loosely it returns the wrong answer with a normal 200 OK and nothing in your logs. So the real work in a semantic cache is not building it. It is choosing the similarity threshold by the cost of a wrong answer, and scoping the cache key so one customer never gets another customer's reply.

Semantic caching is not prompt caching

Both get pitched as "cache your way to a smaller AI bill," and operators conflate them because they share a word. They are different mechanisms with different savings and different risks.

Prompt caching is a provider feature. When you send the same long prefix on many calls, a shared system prompt, a policy document, a set of few-shot examples, the provider stores that prefix and bills the repeated part at a discount. Anthropic reads cached input at roughly one tenth the normal input price. The important part: you still make every call, you still wait for it, and you still pay full price for the output tokens. Prompt caching cannot return a wrong answer, because it is not deciding anything. It is just a discount on input you were going to send anyway.

Semantic caching is your own layer, sitting in front of the model. Each incoming request is turned into an embedding, a vector that captures its meaning, and compared against the embeddings of requests you have already answered. If the closest match is above your similarity threshold, you return the stored answer and never call the model. If nothing is close enough, you call the model, answer the request, and store the new answer for next time.

	Prompt caching	Semantic caching
Who runs it	The AI provider	You, in front of the model
What it saves	Input tokens on the repeated prefix	The entire call, input and output
Trigger	An identical prefix	A request similar in meaning
Can it be wrong	No	Yes, on a false match
Needs an embedding model	No	Yes

They stack. A support automation can run prompt caching on its system prompt and semantic caching on its incoming questions at the same time. The distinction that matters: prompt caching is free money with no downside, so turn it on. Semantic caching trades a real correctness risk for bigger savings, so it needs a decision.

The similarity threshold is a wrong-answer dial, not a speed setting

Here is the point most build guides skip. The similarity threshold looks like a performance tuning knob, the way you would tune a cache size or a timeout. It is not. It is the false-answer rate you are agreeing to serve.

Lower the threshold and more requests count as a hit, so your hit rate and savings climb. But a lower bar also lets in requests that are similar in wording and different in meaning, and those return a stored answer that is simply wrong. In one published test set (AWS figures, reported by the AI gateway vendor Portkey), the tradeoff looked like this:

Threshold	Hit rate	Cost savings
0.99	23.5%	15.8%
0.95	56.0%	51.9%
0.90	74.5%	72.5%
0.80	87.6%	84.6%

Read that as a temptation. Dropping from 0.95 to 0.80 nearly doubles the hit rate and pushes savings past 84 percent. On a benign, general question set the answer accuracy in that test barely moved. But accuracy is a property of your data, not of the threshold. In a domain where a small wording change flips the meaning, "cancel my subscription" versus "can I cancel my subscription," the same 0.80 threshold that was safe for trivia starts serving confident wrong answers.

So the number is a product decision, not an engineering one. Set it by asking what a false hit costs in this specific flow:

• A cache in front of store hours, return policy, or "what does this feature do" can run loose. A wrong answer is cheap and rare, and the savings are large.
• A cache in front of anything with money, eligibility, health, or legal meaning belongs at 0.97 or higher, barely more permissive than an exact match, where you are only catching obvious rephrasings.
• If the honest answer is "a wrong reply here is unacceptable," do not semantic-cache that request at all. Use an exact-match cache or no cache.

The practical ceiling from real deployments: once your false-hit rate climbs past roughly 3 to 5 percent, you have hit the limit of your embedding model, and lowering the threshold further just buys hit rate with wrong answers. A better embedding model raises that ceiling more than a looser threshold ever will.

In a shared automation, an unscoped cache key leaks answers

The second failure mode is the one that turns a cost optimization into a privacy incident, and it is specific to automations that serve more than one customer from the same cache.

If your cache key is only the meaning of the request, then two customers who ask the same question hit the same cache entry. That is exactly what you want for "what are your hours." It is a data breach for "what is the status of my order" or "how much do I owe." Customer A asks, the model answers with A's private data, and the answer gets stored keyed on the meaning of the question. Customer B asks the same question, matches the stored embedding, and receives A's order status. No error fires. Both requests returned 200 OK.

The fix is to scope the cache key. The key is not just the request embedding. It is the embedding plus the identity boundary the answer depends on: the customer ID, the account, the tenant. Two customers asking the same account-specific question now land in different cache namespaces and never collide. Requests that are genuinely shared, the policy and product questions, stay in a common namespace and keep their high hit rate.

The rule is blunt: if the answer to a request depends on who is asking, the identity has to be part of the key, or that request does not go in a shared cache. This is the same discipline as keeping one customer's data out of another's logs. If you already treat automation logs as a PII surface, the cache is one more store of the same data and needs the same scoping.

Where semantic caching does not belong

A semantic cache is worth adding only where the answer to a repeated question does not change. Several common request types quietly fail that test, and caching them ships stale or wrong data even at a perfect threshold:

• Anything with a live value: prices, inventory counts, balances, appointment availability. The stored answer is right the moment you cache it and wrong an hour later. Cache the shape of the response, never the live number.
• Anything personalized that you cannot cleanly scope: recommendations, account summaries, anything blending public and private context in one answer.
• Anything already deterministic. If the request maps to a fixed answer, a lookup table or an exact-match cache is cheaper, faster, and cannot false-hit. You do not need embeddings to return a canned reply to an exact string.
• Steps where the model output feeds a downstream branch or a dedup key. A cached answer that is slightly off can silently change which path the automation takes.

The best-fit case is high-volume, read-only, general questions where many users ask the same thing different ways: a support FAQ bot, a documentation assistant, an internal knowledge lookup. That is where the embedding match earns its keep, catching "how do I reset my password" and "I forgot my login" as the same question, which an exact-match cache would miss entirely.

How to add it without shipping a silent bug

Treat the cache as a change that can fail invisibly, and instrument it before you trust it.

1. Turn on provider prompt caching first. It has no downside and needs no similarity decision. Confirm your long prefixes are actually getting cache hits before you build anything more. Our note on when prompt caching actually pays off covers the traps there.
2. Pick one high-volume, read-only request type. Do not cache the whole automation. Cache the one endpoint where the same general question arrives constantly.
3. Start the threshold at 0.95 and scope the key by customer or tenant from day one, even in a prototype. Retrofitting the scope after a leak is worse than building it in.
4. Log every cache hit, and sample a fraction back through a live model call. Compare the cached answer to the fresh one and track the disagreement rate. That number is your real false-hit rate, and it is the only way to see the silent failures.
5. Only then lower the threshold, watching that disagreement rate, and stop the moment it crosses the ceiling you set for that flow.

The build options are mature. GPTCache is the canonical open-source semantic cache and integrates with the common orchestration frameworks. Redis offers a productized semantic cache, and AI gateways like Portkey bundle it as a setting. The tool is not the hard part. The threshold and the key are.

Semantic caching is a real lever, often a larger one than switching models, because it removes calls instead of making them cheaper. It composes cleanly with a two-tier model cascade: cache the repeats, then cascade whatever misses the cache. But it is the one cost optimization on this list that can return a wrong answer, so it earns the same rigor you would give any step that talks to customers. If you want a second set of eyes on where a cache fits in your stack and where it would bite, tell us what the automation does and we will map it.