Your Semantic Cache Answers the Question Next Door

We replayed 288 questions to an ops assistant through a semantic cache. At a similarity threshold of 0.80 with bge-m3, the cache answered 32 percent of them from memory and cut the average response time from 8.7 seconds to 6.6. It also answered about one hit in three with the answer to a different question.

Someone asked how to undo a commit they had already pushed to main. The cache had seen "how do I undo my last commit that I have not pushed yet", decided the two were the same question, and served the answer for a local commit: git reset --soft HEAD~1. For a commit already on main the right answer is git revert; resetting it and pushing the result means force-pushing over history other people have pulled.

Raising the threshold does not fix it. Between 0.88 and 0.92 the few hits left were more often wrong than right. Across six embedding models and thresholds from 0.50 to 0.99 in steps of 0.01, the best any of them managed with no wrong answers at all was a 0.7 percent hit rate. For four of the others, the only thresholds that never served a wrong answer never served anything, and one model served a wrong answer even at 0.99.

This post covers how we measured it, why no threshold separates the two kinds of question, what the fix everyone reaches for actually does, and where a semantic cache is genuinely safe.

TLDR

• The setup. 24 pairs of ops questions that read almost the same and need different answers: restart versus reload nginx, the staging versus the production password, a memory limit versus a memory request. Six phrasings each, 288 questions, labelled by construction, so a wrong hit is a fact and not a judge model's opinion.

• The cache. Embed the question, find the nearest one already answered, serve its answer if the cosine similarity clears a threshold. Six embedding models on DigitalOcean Serverless Inference, 50 thresholds, 20 orderings of the questions.

• At 0.80 with bge-m3: 32 percent hit rate, 62 right hits and 30 wrong ones on average per run of 288 questions.

• From 0.88 to 0.92: more of the remaining hits were wrong than right.

• The same threshold is not the same setting on another model. At 0.80, e5-large-v2 answered 88 percent of questions from the cache and 65 percent of those hits were wrong.

• What it saved: on gpt-oss-120b at published prices, $0.25 per 1,000 questions without a cache and $0.18 with one at 0.80. The average got faster; the slowest 5 percent barely moved.

• Where it is likely safe: near-verbatim repeats. Most of them scored higher against their original than any near-miss pair did, so a threshold set from your own near-misses should catch them. We did not replay them through the cache.

• The obvious fix works, and costs more than it saves. A small model checking each hit before it was served cut wrong answers to under one per run on average, at every threshold. It also left the cache no faster than having no cache, and more expensive at every threshold tested.

Prerequisites

• Node.js 20 or later. The harness has no dependencies.
• A DigitalOcean Serverless Inference key. The docs cover creating one.
• Patience for the embedding stage: 288 questions times six models, one request each, under a per-minute rate limit. The simulation stages after that cost no inference at all.

How a semantic cache decides

An exact-match cache only helps when the same text arrives twice, perhaps after normalising case and punctuation. A semantic cache is meant to help when the same question arrives in different words:

1. Turn the incoming question into an embedding.
2. Find the most similar question already answered.
3. If the similarity is above a threshold, return that question's stored answer.
4. Otherwise ask the model, and store the new question and answer.

The threshold is the only thing deciding whether two questions are "the same". It is one number, compared against one similarity score, and it has to work for every question the assistant will ever get. That is the part worth testing.

The questions

Operations questions come in near-miss pairs all the time. The words barely change and the right answer changes completely, often in the direction that breaks something.

Pair	Why the answer differs
restart nginx / reload nginx	A restart stops the process and can drop connections; a reload applies config gracefully
rotate the staging password / the production one	Same operation, different blast radius and procedure
memory limit / memory request	A limit causes OOM kills; a request affects scheduling
delete a pod / delete a deployment	A pod its deployment owns comes back; the deployment does not
undo a pushed commit / an unpushed one	Revert is safe on shared history; reset rewrites it
terraform state rm / destroy one resource	One forgets the resource, the other deletes it
clear one Redis database / all of them	FLUSHDB versus FLUSHALL
cordon a node / drain it	One stops new pods; the other evicts the running ones

There are 24 pairs like these, each side written six different ways, from "How do I restart nginx?" to "How can I bounce the nginx process completely?". Two questions count as sharing an answer if they are listed under the same intent, and a hit counts as wrong when the stored answer was written for the other intent of the pair, or for an unrelated one. No model grades it.

This is a stress test, not an estimate of how often a production cache is wrong. Every question's near-miss partner is in the stream, there are no exact repeats, and the paraphrases were written to vary their wording while the near-miss pairs keep theirs. All of that makes it hard on a cache on purpose. How many near-misses your own traffic contains is something only your logs can tell you. What this measures is what the cache does when they turn up.

Why one threshold cannot separate them

Here are two kinds of question pair, scored by bge-m3. The green curve is every pair that needs the same answer. The red dashed curve is every designated near-miss pair: the partners that read almost the same and need different answers. Unrelated pairs, the other 96 percent of the 41,328 possible, are left out.

A threshold is a single vertical line through both curves. A real cache only compares a new question against what it has stored, so this is not a hit rate. But it shows the problem: no line sits to the right of the whole red curve without also sitting to the right of almost all of the green one. The two overlap across nearly their whole range.

The overlap was large for every model we tried:

Embedding model	Median, same answer	Median, near-miss	Most similar near-miss	Near-miss pairs above the median paraphrase
bge-m3	0.723	0.646	0.932	22.8%
gte-large-en-v1.5	0.771	0.707	0.958	27.1%
qwen3-embedding-0.6b	0.756	0.685	0.949	22.6%
multi-qa-mpnet-base-dot-v1	0.684	0.577	0.927	23.5%
e5-large-v2	0.861	0.837	0.982	36.2%
all-mini-lm-l6-v2	0.849	0.795	0.991	36.2%

For at least one near-miss pair in five, the two questions that need different answers look more alike to the embedding model than a typical pair of questions that need the same one. Embedding models are trained to put questions about the same topic close together, and near-miss pairs are exactly that: the same topic, one word apart.

The sweep

We replayed the 288 questions through the cache at every threshold from 0.50 to 0.99, in 20 different random orders. The order matters, because which phrasing arrives first decides what the cache stores and what later questions get matched against. Each threshold gets its own replay, since a question that hits at 0.80 is not stored, while the same question at 0.90 misses and is.

At the low end the cache hits often and is wrong often. As the threshold rises, right hits fall away faster than wrong ones, because the near-miss pairs share their wording and the paraphrases, on purpose, do not. From 0.88 to 0.92, the few hits that remained were more often wrong than right. At 0.93 there was one of each, and above that none at all.

The same number means something different on each model. At 0.80:

Embedding model	Hit rate	Right hits	Wrong hits	Share of hits that were right
bge-m3	32.0%	62.2	29.9	68%
gte-large-en-v1.5	52.7%	100.8	50.9	66%
qwen3-embedding-0.6b	44.8%	83.4	45.7	65%
multi-qa-mpnet-base-dot-v1	28.1%	56.6	24.4	70%
e5-large-v2	88.0%	88.8	164.6	35%
all-mini-lm-l6-v2	76.5%	93.4	126.9	42%

A threshold is only meaningful next to the model that produced the scores. Advice like "use 0.85" without naming the embedding model is not advice.

What a wrong hit looks like

This is npm run show-wrong-answers, which replays one ordering and prints every question that was answered with another question's answer:

For 12 of the pairs we marked which answer does more damage when it is served for the other question. Those labels are judgment calls and they are in the repo so anyone can argue with them. At 0.90 on bge-m3, which is a high setting for that model:

bge-m3 at 0.90: 10.0 near-miss wrong answers per run, 2.5 of them the more destructive answer
    asked rotate-db-password-production  got rotate-db-password-staging     in 17 of 20 runs (21 times)
    asked cdn-purge-one                  got cdn-purge-all                  in 12 of 20 runs (12 times)
    asked ssh-rotate-user-key            got ssh-rotate-host-key            in 10 of 20 runs (10 times)
    asked docker-stop                    got docker-kill                    in 6 of 20 runs (6 times)

Someone asking how to rotate the production database password got the answer written for staging in 17 of the 20 runs. The recorded answers show why that matters: none of the six staging answers mentions avoiding downtime, and three of the six production answers do.

What it actually saves

Every question now pays for an embedding call, hit or miss. Every miss still pays for the full answer. These figures are built from API timings and token counts measured on DigitalOcean one question at a time, then replayed through the cache; they are not timings from a deployed cache:

	Mean per question	p50	p95	Cost per 1,000 questions
No cache	8.66 s	8.33 s	14.53 s	$0.2534
bge-m3 at 0.70	3.48 s	0.53 s	13.14 s	$0.0871
bge-m3 at 0.80	6.56 s	7.03 s	14.54 s	$0.1775
bge-m3 at 0.90	8.74 s	8.69 s	15.05 s	$0.2414
bge-m3 at 0.95 (nothing hits)	9.17 s	8.89 s	15.05 s	$0.2537

Three things stand out.

The money is small. On gpt-oss-120b at DigitalOcean's published prices, answering 1,000 of these questions costs about 25 cents. At 0.80 the cache saves 7.6 cents of that, and serves about 104 wrong answers per 1,000 questions to do it. Embedding every question cost a fraction of a cent across the whole run; the cache's overhead is time, not money.

The average improves, the tail barely does. A miss now costs an embedding call plus the answer. At 0.80, p95 was about 14.5 seconds with the cache and without it. It falls only at lower thresholds: at 0.70, p95 was 13.1 seconds, and 39 percent of the hits were wrong.

A cache that never hits is pure overhead. At 0.95 and above, nothing hit and every question paid about half a second extra for its embedding.

The fix everyone reaches for

If the embedding cannot tell restart from reload, ask a model that can. Before serving a hit, send both questions to a small, cheap model with one instruction: would exactly the same answer be correct for both? Serve the hit only if it says yes.

We ran that with gpt-oss-20b as the verifier, in front of bge-m3, over the same 20 orderings. Every candidate hit was checked; 1,162 distinct question pairs in all.

Threshold	Without verifier: hit rate	Wrong	With verifier: hit rate	Wrong	Right hits it rejected	Mean per question	Cost per 1,000
0.70	67.2%	76.1	40.1%	0.3	29.6	8.57 s	$0.2773
0.75	52.8%	53.1	32.0%	0.5	21.9	8.59 s	$0.2679
0.80	32.0%	29.9	21.2%	0.5	9.9	8.70 s	$0.2577
0.85	15.2%	16.4	9.8%	0.5	3.0	8.91 s	$0.2540
0.90	4.9%	10.0	1.2%	0.5	1.0	9.22 s	$0.2569
no cache						8.66 s	$0.2534

It works. Wrong answers dropped from as many as 76 per run to under one on average, and never more than two in any run, at every threshold we tried. At 0.70 the verified cache still served 40 percent of questions from memory. The only two pairs it let through were both memory request against memory limit. It erred the other way far more: it turned down 146 of the 701 genuine paraphrase pairs it was shown, which is where the lost hits went.

And it defeats the point. The mean response time with the verifier, 8.6 seconds at 0.70, is the same as with no cache at all. Each check took 3.2 seconds at the median, and it runs on every candidate hit, including the ones it then rejects, which also pay for a fresh answer. The cost went up too: $0.28 per 1,000 questions at 0.70, against $0.25 with no cache.

The reason is in the token counts. The verifier replies with one word, same or different, and is billed for a median of 283 completion tokens to produce it, because it is a reasoning model and reasons first. The answers it was saving had a median of 324. At the smaller model's lower price, a median check still cost about 57 percent of a median answer, and it runs more often than there are hits to save.

A small non-reasoning model could be faster and cheaper per check. We did not test one, and it would have to be just as reliable on exactly the pairs where the embedding models failed.

Where a semantic cache is safe

The corpus above is built from real paraphrases: different words, same question. Real traffic also contains plenty of near-verbatim repeats, the same question again with trivial differences. We made one of those for every question: lower case, no question mark, and sometimes a prefix such as "hey," or "quick one:", and compared how similar each question is to its own repeat against how similar the near-miss pairs get.

Embedding model	Lowest-scoring repeat	Most similar near-miss	Repeats scoring above every near-miss
bge-m3	0.856	0.932	244 of 288
gte-large-en-v1.5	0.882	0.958	258 of 288
qwen3-embedding-0.6b	0.848	0.949	210 of 288
multi-qa-mpnet-base-dot-v1	0.866	0.927	274 of 288
e5-large-v2	0.945	0.982	186 of 288
all-mini-lm-l6-v2	0.924	0.991	225 of 288

For every model, most trivial repeats scored higher against their own original than any near-miss pair in the corpus scored against each other. That suggests a threshold set just above your worst near-miss would catch most such repeats without serving a near-miss. We did not replay the repeats through the cache, and a repeat could still score higher against some other question, so treat it as a likely saving rather than a measured one. It is also much closer to an exact-match cache than to the semantic one people have in mind. Case, punctuation and a greeting are exactly what a normalised exact-match key would catch, with no embedding call. Normalisation has risks of its own and we did not measure that comparison, but it is the baseline a semantic cache has to beat for this kind of repeat. And the high threshold only works if you have measured your own near-miss pairs to find where "just above" is.

What we could not conclude

• How often near-misses happen in real traffic. This corpus puts every question's near-miss partner in the stream, which is a deliberately difficult case for a cache. The hit rates and wrong-answer counts depend on how repetitive your traffic is and how many near-misses it contains, and only your own logs can say.
• Whether every counted wrong answer was actually wrong. Hits are scored against our intent labels, not by reading each answer, and the labels are not perfect. "Empty Redis database 2 only" sits in the same intent as "clear only the current database", though it needs one more command. We read the damaging examples in this post by hand; we did not grade all of them.
• Other cache designs. This is one global cosine threshold over the question alone, with nearest-neighbour lookup. Partitioning the cache by environment or resource, or only caching some kinds of answer, would behave differently, and we did not test them.
• Anything beyond ops questions in English. 24 pairs, one domain, one language.
• How much of this is the embedding models and how much is the corpus. We wrote the paraphrases to vary their wording and the near-miss pairs to share theirs. A corpus with closer paraphrases could give higher hit rates; we did not test one.
• Statistical intervals. The 20 orderings are replays of one fixed set of questions, not independent samples, so the ranges we report are spread across orderings, not confidence intervals.
• Vector database latency. The in-memory search measured 1 to 3 ms. A vector store across the network would add its own round trip, which we did not measure.
• Whether a wrong answer would have been acted on. We counted wrong answers served, not wrong actions taken.

What we would do

1. Build your own near-miss set before you pick a threshold. Write down the questions your users ask that differ by one word and need different answers. Score them with your embedding model. Your threshold has to sit above the most similar of those pairs, and that number is specific to the model and to your domain.
2. Treat a threshold as a property of a model. Changing the embedding model without re-measuring changes what the cache does, sometimes from mostly right to mostly wrong at the same setting.
3. Keep answers that act on things out of the cache. An explanation of what a readiness probe is can be served twice. A command to run against production should not come from a question that merely looked similar.
4. Put the key facts in the key. The expensive mistakes here were a word apart: staging or production, one database or all of them, pushed or not. If the environment, the resource and the verb are part of the cache key, the similarity search never gets the chance to confuse them.
5. Measure the whole trip. The embedding call is paid on every question, and it makes every miss slower. Count it before deciding the cache saves anything.
6. If you add a verifier, price it. Checking each hit with a model made the cache correct here, but no faster and dearer than no cache, because the verifier reasoned its way to a one-word answer. Measure it on your own near-miss pairs and count its tokens before you ship it.

The harness, the 288 labelled questions, every embedding and answer, and the scripts that reproduce every number here are in the repo.