Caching architecture for low‑latency LLMs

A large language model feels slow for the same reason a restaurant kitchen does: if every order starts from raw ingredients, every customer waits. Caching changes that by saving work the system has already done and reusing it on the next similar request. (developers.openai.com) The simplest cache is an exact-match cache. If a user sends the same prompt twice, a key-value store like Redis can return the saved answer in milliseconds instead of sending the request back through the model. (redis.io) That only solves the easy case, because people rarely ask the same thing with the same words. A semantic cache turns each prompt into a numeric fingerprint called an embedding, then looks for old prompts that are close in meaning rather than identical in spelling. (redis.io) The dangerous part is the similarity threshold. If the threshold is too low, “reset my password” can wrongly match “change my email,” and the system returns a fast but incorrect answer; if the threshold is too high, the cache misses too often and the model bill stays high. (docs.redisvl.com) That is why the best production designs use tiers instead of one giant yes-or-no cache. The first tier checks for an exact hit, the second checks for a high-confidence semantic hit, and only then does the request go to retrieval or a fresh model call. (docs.litellm.ai) A vector database is the part that makes semantic lookup fast. It stores those numeric fingerprints and can search millions of prior prompts for near matches the way a music app finds songs that sound alike. (redis.io) Redis often shows up twice in these stacks because it does two different jobs well. It can act as a classic in-memory cache for exact hits and, with vector search, as the fast layer for semantic hits too. (redis.io) The recent write-up making the rounds pushes that idea into a full enterprise pipeline: route each request through graded checks, attach metadata like model name and tenant, set expiration times, and log every hit, miss, and fallback. That turns a vague “the model seems expensive today” problem into a measurable system with rules. (x.com) Kubernetes enters the picture because caches stop being simple when traffic spikes. If one container has a hot cache and another new container has nothing, users get random latency, so teams use shared cache layers and autoscaling to keep response times stable across replicas. (docs.litellm.ai) Observability is the part most demos skip and most real systems need. Teams track cache-hit rate, semantic-match score, token savings, wrong-answer reports, and time-to-first-token so they can see whether a faster answer was also the right answer. (docs.redisvl.com) There is also a second kind of caching that happens inside the model provider itself. OpenAI’s prompt caching automatically reuses repeated prompt prefixes on recent models, which means a long system prompt or repeated context block can get lower latency and lower input cost even before your own application cache runs. (developers.openai.com) Put together, the architecture is less about one clever trick than about deciding which work deserves a full model call. When exact matches, semantic matches, provider-side prompt reuse, and careful logging all sit in front of the model, the expensive part becomes the exception instead of the default. (developers.openai.com)