Semantic caching, explained: when an AI can reuse an answer

You ask a lamp shop for a refund on order 4417. An answer comes back at once. Then you notice the order number in the reply: 4471. It belongs to somebody else.

The shop is fictional; the mistake isn’t. Its support assistant had saved its answer to an earlier refund question. The software decided your question was close enough to that one and sent the old reply. That shortcut is called a semantic cache. Used well, it makes an assistant faster and cheaper. Here’s how it works on an everyday question, and how it goes wrong when two digits change.

The shortcut: reuse an answer you already have

The shop’s delivery policy is simple: standard delivery anywhere in Ireland takes 3 to 4 working days. On a Monday morning, three customers ask about it in different words. Without a cache, the assistant sends each question to an AI model, waits a few seconds, and gets back much the same answer each time.

A cache stores answers you’ve already worked out. The simplest kind only reuses an answer when the new question is character-for-character identical, which people almost never type. A semantic cache compares meaning instead, so “How many days to ship to Dublin?” can reuse the answer to “How long does delivery to Ireland take?”.

How a computer measures “close in meaning”

An embedding is a list of numbers that an embedding model (a small, cheap AI model) produces for a piece of text. Think of it as a position on a map of meaning: questions about the same topic land close together. Real embeddings have hundreds of dimensions rather than two, but the idea is the same.

The cache then works out a similarity score: here, a number from 0 to 1 that says how close two questions are, by that model’s measure. A score of 0.97 isn’t a 97% chance the answer is right; it only says the questions are near each other on the map. If the score is above a threshold you choose, the cache reuses the stored answer. If not, it asks the model and stores the new answer.

One morning at the lamp shop

1. 9:02, “How long does delivery to Ireland take?” The cache has nothing close, so it’s a miss. The model answers “3 to 4 working days”, and the cache stores the question and the answer.
2. 9:15, “How many days to ship to Dublin?” It scores above the threshold against the 9:02 question. A hit, and a right one: the policy covers all of Ireland.
3. 9:40, “How many working days should I allow for delivery to Cork?” Same question again, in longer words, and it scores just under the threshold. A miss: the stored answer would have been right, but the model is asked anyway.
4. 9:55, “Do you deliver to Cork?” Same topic, so it scores above the threshold. A hit, and the wrong kind: the customer asked whether the shop delivers there and got how long it takes.
5. 10:05, “Can I get a refund on order 4417?” Someone asked about order 4471 earlier, and this is the highest score of the morning. A hit, with somebody else’s answer.
6. 14:00, the shop changes courier. Delivery now takes five working days. Every stored delivery answer is wrong, and nothing in the cache knows.

The threshold can’t fix this

The obvious fix is a stricter threshold. Put the morning’s scores side by side and it stops looking obvious.

The score measures how close two questions are in topic. Whether a saved answer is still correct depends on details the score barely sees: which order, which fact the customer wants, and what day it is. So the useful question isn’t “do these mean the same?” but “would the saved answer still be right for this customer, about this order, today?”

A rule for what the cache may answer

• General policy questions (delivery times, returns, how to fit a bulb) can come from the shared cache, as long as the answer is the same for everyone.
• Order questions look up that exact order, after checking the customer is allowed to see it. Putting the customer in the cache key doesn’t solve the 10:05 mistake: one customer can own both orders.
• Refunds and other actions never come from a cache. Check what has already happened, then act or confirm, so a repeated question never issues a second refund.
• Expiry. Give every stored answer a lifetime, and clear the delivery answers when the policy changes, as it did at 14:00.
• One business, one cache. If several companies use your product, their answers never mix.

What it actually saves

Every question now pays for an embedding, which costs very little (OpenAI lists text-embedding-3-small at $0.02 per million tokens). A hit saves the chat-model call. With a small model like GPT-6 Luna, at the $0.10 per million input tokens and $0.50 per million output tokens my experiments used, a reply with 500 tokens in and 300 out has a modelled cost of about $0.0002 at list prices.

So a hit saves a few hundredths of a cent with a small model. That adds up at high volume or with an expensive model, and the seconds a hit saves often matter more than the money. Set both against the cost of one wrong answer.

Before you switch one on

• Decide which kinds of question the shared cache may answer, and which are always looked up.
• Set an expiry, and a way to clear answers when a policy changes.
• Test near-misses before going live: questions that differ only in a number, a date or a name, and questions on the same topic that ask for a different fact.
• Log every hit with both questions, and have someone read a sample each week.

Limits

• The similarity scores in this post are illustrative. Real scores depend on the embedding model, and a threshold that suits one model means little for another.
• This is about caching answers to whole questions. “Prompt caching”, which some AI providers offer to reuse the start of a long prompt, is a different thing with different risks.
• The lamp shop is fictional; the failure patterns are common ones.

My next experiment puts real numbers on this: how often questions like the 9:55 and 10:05 ones score above a threshold, on two embedding models.

First published 10 Nov 2026.