Skip to main content

Response cache

Updated on 13 November 2026.

Answer a repeat request from the response the gateway already delivered, without calling the model again. A cache hit costs nothing and skips the model's response time. You choose how closely a new request must match a stored one: an exact repeat, a repeat that differs only in case, punctuation or filler words, or a question asked in different words.

The response cache is off by default and is separate from a provider's prompt cache. To keep the provider's prompt cache alive instead, see Keep prompt cache warm.

What it caches​

RequestCached
OpenAI Chat Completions, non-streamingYes
Anthropic Messages, non-streamingYes
EmbeddingsYes, exact match only
Streaming requests and other APIsNo, always sent to the model

A stored response is reused only for the same app, model, provider, settings and policy. With the user scope, each person also gets their own entries. Entries expire after the app's cache lifetime (24 hours by default).

Responses that were blocked, or where a guardrail could not finish its check, are never stored.

Match modes​

ModeA new request matches whenWorks on
Exact (default)It is the same request. In prompt text, only line endings, trailing spaces on each line, leading and trailing whitespace, and Unicode forms are cleaned up first.Chat, Messages, embeddings
NormalizedIt is the same after also ignoring case, punctuation at the edges of words, repeated spaces and a fixed list of words (below). Applies to all prompt text, including the system prompt and earlier messages.Chat, Messages
SemanticIts latest user message asks the same thing in different words. Everything else must match exactly.Chat, Messages

Normalized ignores exactly these words: a, an, the, is, are, am, be, do, does, i, me, my, please, kindly, pls, plz, hi, hello, hey, dear, thanks, thank, regards, just. So "Can I access my records?" and "Can I access the records?" are treated as the same request.

Each mode builds on the one before it: Semantic first tries a normalized match, and Normalized covers every exact repeat.

Lookup order​

  1. Is the same request already stored?Exact, or Normalized in Normalized and Semantic mode
    yes:
    Cache hitStored response returned. Skips every guardrail and the model call
  2. Do request guardrails block the new prompt?Data risks, Guardian Agent and other request checks run on the new prompt
    yes:
    BlockedSame result as without the cache
  3. Semantic mode: was a close enough question already answered?Everything but the latest user message matches, prompt not redacted or flagged, checks pass
    yes:
    Semantic hitStored response returned. Skips the model call and response guardrails
  4. None of the above
    Sent to the modelThe response is stored for next time
QuilrAI

Exact and normalized repeats are answered straight from the cache and skip every guardrail, in Semantic mode too, because that check comes first. In Exact and Normalized mode a miss goes through the normal pipeline to the model.

How semantic matching stays safe​

  • Only the latest user message can differ. Everything else (system prompt, earlier conversation, tools, model, settings and policy) must match exactly.
  • Request guardrails run first. A reworded match is served only after request guardrails (data risks, Guardian Agent) pass the new prompt. It skips the model call and the response guardrails. A prompt that request guardrails blocked, redacted or flagged is never matched semantically.
  • Simple checks on the two latest user messages, compared in lowercase after normalization:
    • numbers, links and email addresses must be the same;
    • text in backtick code spans must be the same;
    • negation words (not, never, can't and similar) must match;
    • the two messages must be of similar length.
  • Fails open. If the similarity check is unavailable or slow, the request goes to the model.

Similarity​

Similarity sets how close a reworded question must be. It applies only in Semantic mode.

SimilarityMatches
Narrow (default)Near-identical wording. Safest.
BalancedLight rewording.
WiderLooser rewording. More hits and more risk of a wrong answer.

Results from simulated real-world traffic​

In an offline replay of about 41,000 requests of simulated real-world traffic across 63 applications over one week, exact repeats accounted for about 99% of the cache hits that exact and normalized matching found, and normalized matching for about 1%. Turning on Semantic matching adds more hits:

SimilarityExtra hits from semantic matching
Narrow~5%
Balanced~8%
Wider~12%
Semantic matches can match unexpected prompts

Prompts that differ only in a small detail can be treated as the same question, so the cache returns an answer generated for something else. For example:

Prompt cached firstLater prompt that matched it
Write a welcome email for new hires at Acme CorpWrite a welcome email for new hires at Globex Inc
Summarize the duties of a Senior Data EngineerSummarize the duties of a Junior Data Engineer
Check the DNS records for example.comCheck the DNS records for example.org

The checks above do not catch names.

Which mode to choose​

  • Exact or Normalized covers almost all of the benefit. Use them for automated pipelines and agents.
  • Use Semantic only if you want the last few percent, and only for apps where people type questions, such as FAQ or helpdesk. Start with Narrow and review the cache hits in the request logs.

Configure the response cache​

Choose the mode used by your gateway. When the Policy Engine is on, published rules decide whether the cache is on and which match mode and similarity apply; without a matching rule the cache is off.

  1. Open Govern > Policy Engine > LLM Gateway > Controls > Token Savings.

  2. Start from a recipe: Turn on response cache (Exact), Cache near-identical repeats (Normalized) or Cache reworded questions (Semantic). Or add a Response cache rule and turn it on.

  3. Choose Applies to. Under Cache matching pick Exact, Normalized or Semantic. For Semantic, add Semantic similarity and pick Narrow, Balanced or Wider. Choosing Semantic shows the wrong-answer warning.

  4. Save to draft, use Test, then Review & publish.

The same rule in policy text:

language 1;
policy helpdesk_cache priority 600 {
request where application.group_key == "helpdesk"
then { response_cache.enabled = true; response_cache.match = semantic; response_cache.similarity = narrow; }
}

response_cache.match takes exact, normalized or semantic; response_cache.similarity takes narrow, balanced or wider. Without a matching rule they are exact and narrow. Conditions on response_cache.enabled, .match and .similarity cannot use data types, about(...), Guardian Agent, hallucination or language results, or the final routed model fields, because the cache decides before those are known.

To set the cache from your own tooling, use /controls/response-cache in the Access and gateway features API with the settings enabled, match and similarity.

Check that it works​

Supported responses that reach the cache step carry an X-Quilr-Cache header:

HeaderMeaning
X-Quilr-Cache: HITServed from the cache, including semantic hits. Age gives the entry's age in seconds.
X-Quilr-Cache: MISSNo cached response was served; normal processing continues (the request can still be blocked later). The response may be stored for next time.
X-Quilr-Cache: BYPASSNot looked up, for example a request with Cache-Control: no-store or no-cache.

A hit is logged with zero tokens and zero cost, and records the tokens and cost it saved. Hit rate and savings per app are in the Monitoring API feature summary.