How the two-layer caching of Prompt Caching and Semantic Caching saves LLM costs

Two-layer caching to save costs: Prompt Caching reuses prefix computation, Semantic Caching reuses the entire answer

From @samuel_wong_ posted on @bytebytego, focusing on two types of caching in LLM applications: Prompt Caching reuses the KV computation of a shared prefix, while Semantic Caching reuses the entire answer for semantically similar questions. The former reduces input costs, while the latter may skip the model call entirely. The two can be stacked, but the threshold, invalidation, and scope of applicability for semantic caching must be designed separately. https://t.co/d0goDAJHuO

Why cache: The cost of a single request comes from input tokens and output tokens. In customer service applications, system instructions, product manuals, and conversation history are often far longer than the user's question. Repetition occurs in two places: the same long document is sent to the model countless times; and many users ask the same thing in different wording. The former corresponds to Prompt Caching, and the latter to Semantic Caching.

Prompt Caching: Reusing input processing · The matching rule is that the beginning of the input must be exactly identical. The model first does prefill, then decoding; the KV state of the prefix can be reused across requests. · Example: two requests share "support instructions + product manual," with only the question differing. After the first is processed, the second can directly reuse the prefix's KV. · Shared content must be placed in the prefix. In a causal model, later content changes the representation of earlier text; a timestamp or request ID, even if placed only at the beginning, will break reuse for the entire segment. · Instructions must be literally identical; wording, document order, and request settings must not change. · Costs do not drop to zero: cache writes are usually more expensive, and reads are cheaper. The article cites Anthropic's standard pricing: cache reads are about 1/10 of normal input; a 5-minute write is 1.25x, and a 1-hour write is 2x. For example, if the prefix originally costs $1, then one write + one read = $1.35, not $2; the more it is read, the more worthwhile it is, while never reusing it means a loss. · It mainly saves input processing and time-to-first-token, not the work of generating long answers.

Semantic Caching: Reusing answers · Usually done at the application layer or as a separate service: use an embedding model to turn the question into a vector, retrieve similar historical Q&A; if there is a hit, return the cached answer directly; if not, call the LLM and possibly write to the cache. · On a hit, the entire LLM call is skipped; on a miss, an extra layer of embedding and retrieval cost is paid. · The difficulty is the similarity threshold: strict means a low hit rate, loose may mean wrong answers. Examples of similar but not reusable cases include: cancel before shipping vs. cancel after shipping; Basic plan vs. Pro plan; differences in negation words, numbers, dates, and versions. A similarity score is not equal to the probability of correctness; the threshold needs to be evaluated with a labeled set, looking at both the hit rate and the answer appropriateness rate. · Do not look only at the question text. Order status, references, language, permissions, product version, and conversation state must all go into metadata filtering; do not expect the embedding model to infer them from the question itself. · Invalidation strategies: TTL (short, but may still expire before it ends), version numbers (invalidating the whole store is simple but crude; selective invalidation by dependent document is better), and invalidation when instructions/format/model behavior change. Prompt cache invalidation is different: once the prefix changes, the old KV cannot be used from the change point onward; but if the application keeps stuffing old policies into the prompt, the cache itself will not discover that the business rules have changed.

How to use them together: Recommended order: exact-match cache first → then semantic cache → on a miss, go to the LLM, and try to benefit from prefix caching as much as possible. Checks that need to verify the source data version should be placed earlier. Action-type requests (such as "subscription canceled") must never use cached wording in place of the real operation.

Solution selection · Prompt Caching: shared manuals, stable tool definitions, long conversation history, but each question still needs a fresh answer. · Semantic Caching: FAQs, stable explanations; not suitable for strongly personalized scenarios, fast-changing facts, or scenarios requiring fresh creative output.