ZharfAI Team
A support assistant answers, “Employees may expense a laptop up to $1,500.” Minutes later, another employee asks a similar question and receives the cached answer in under 100 milliseconds. The system saved a model call. It also skipped the facts that the second employee belongs to a different subsidiary, uses another currency, and is covered by a policy revised that morning.
The two questions were close in embedding space. They were not equivalent decisions.
Caching can make an AI service faster, cheaper, and less variable. It can also preserve a confabulation, cross an authorization boundary, serve a revoked policy, or give one tenant an answer derived from another tenant’s data. The practical decision is therefore not simply whether to cache. It is which layer may reuse work, which invariants must match, and which requests must always return to current evidence.
This field guide separates provider prompt caching from exact and semantic response reuse, then builds a cache-control architecture around identity, authority, release state, freshness, and consequence. Its central rule is simple: a response-cache hit is a policy decision, not only a performance event.
Teams often discuss caching as if every hit returned an old answer. That is not true. Four mechanisms have materially different risk:
| Layer | What is reused | Does it reuse the final answer? | Primary control |
|---|---|---|---|
| Provider prefix or prompt cache | Computation for an identical prompt prefix, often attention key/value state | No; the model generates a new continuation | Provider isolation, exact prefix identity, retention policy |
| Exact response cache | A previous final response for the same normalized request and system state | Yes | Complete deterministic key, freshness, invalidation |
| Semantic response cache | A previous response for a merely similar request | Yes | Equivalence gate, authorization scope, consequence limit, measured false reuse |
| Retrieval or tool cache | Search results, parsed documents, tool output, or derived features | Indirectly; cached evidence shapes a new answer | Source version, permission, observation time, dependency expiry |
The first layer is frequently the safest optimization because it reuses compute without substituting a previous conclusion. The current OpenAI prompt-caching guide says hits require exact prefix matches and that a new response is still generated. Anthropic’s prompt-caching documentation likewise requires identical cached prompt segments and describes workspace or organization isolation depending on platform. These are provider-specific facts reviewed on the publication date, not guarantees for every model service.
Semantic response caching makes a stronger claim: two different requests are close enough that one answer can stand in for the other. A 2026 research paper, From Exact Hits to Close Enough, explicitly notes that semantic reuse breaks classic cache assumptions. Similarity helps find a candidate; it does not prove that users, permissions, dates, policies, sources, or required actions are interchangeable.
Mature HTTP caching is built around conditions for storage and reuse. RFC 9111 defines cache keys, freshness, validation, Vary, private, and rules for authenticated requests. A stored response is not reused merely because it exists. The request must match the relevant key dimensions, and the response must be fresh, explicitly allowed to be stale, or successfully revalidated.
An AI cache needs the same discipline but a richer notion of identity. A URI and a few headers rarely describe the complete decision state of an AI application. The answer may depend on the user’s role, tenant, locale, current policy corpus, retrieval filter, model release, prompt template, tool schema, safety policy, and the time at which external facts were observed.
The HTTP analogy has a limit. HTTP normally matches declared fields exactly. A semantic cache deliberately relaxes request equality. That relaxation creates a new obligation: the application must define and test decision equivalence, not assume that cosine similarity represents it.
Several facts establish the boundary. Provider prompt caches can reuse identical prefixes without reusing final output. HTTP caching restricts reuse by key, authorization, and freshness. NIST AI 600-1 identifies confident false content, or confabulation, as a generative-AI risk and emphasizes that high-integrity information should have evidence, chain of custody, and an expectation of when validity may expire. OWASP LLM02:2025 warns that LLM application context can contain personal, financial, health, credential, legal, and confidential business information, and recommends strong access control and data sanitization.
The control model below is ZharfAI analysis derived from those sources. It is not a published standard, and a similarity threshold cannot certify correctness. The architecture treats response reuse as an allow-listed decision whose evidence can be inspected, tested, revoked, and measured.
Classify the answer by consequence and volatility. This prevents a vector database from turning every repeated-looking question into a reuse opportunity.
| Workload | Default mode | Why |
|---|---|---|
| Public, versioned product documentation | Exact response cache; semantic candidate after testing | Low personalization; explicit document versions and invalidation are possible |
| Internal policy explanation | Exact cache inside one authorization scope | Policy, jurisdiction, employment class, and effective date can change the answer |
| Personalized account, health, legal, or financial information | No shared response cache | Identity and current records are part of the answer; leakage and stale-state impact are high |
| Live price, inventory, risk, incident, or approval state | Retrieval cache only with short, source-defined freshness | Similar wording does not make time-sensitive facts reusable |
| Consequential recommendation or proposed action | Generate from current evidence; cache only non-decision components | The cost of a wrong reuse can exceed all latency savings |
| Creative ideation with no factual promise | Optional semantic cache for inspiration, visibly labelled | Repetition may be acceptable, but user context and rights still matter |
“No response cache” does not mean “no optimization.” Reuse a safe prompt prefix or deterministic public artifact without pretending an old conclusion is current.
An exact response cache is only exact relative to its key. Hashing the user’s last message is not enough. Construct a canonical identity envelope before lookup:
| Key dimension | Example | Why omission is dangerous |
|---|---|---|
tenant_id and principal_scope | organization, role, entitlement set | A hit can cross customer or permission boundaries |
purpose | public explanation, employee help, analyst review | The same text may be permitted for one use and prohibited for another |
locale and jurisdiction | fa-IR, en-GB, applicable country | Translation, currency, calendar, and legal context can change meaning |
release_id | model, prompt, tools, safety policy | Old behavior survives a supposedly complete deployment |
knowledge_id | corpus snapshot, index build, source versions | The answer can outlive a corrected or revoked source |
retrieval_policy | filters, top-k, reranker, access predicate | Different evidence paths collapse into one key |
output_contract | schema, citation requirement, allowed actions | A free-text answer may be reused where structured evidence is required |
time_boundary | observation time or valid-as-of bucket | Live facts become timeless by accident |
sensitivity | public, internal, confidential, restricted | Storage, encryption, sharing, and deletion rules differ |
Bind release_id to the same assembly described by the AI release passport. A prompt edit, tool-permission change, model alias movement, retrieval rebuild, or safety-policy update should produce a new identity even when the user’s words are unchanged.
Do not put raw sensitive values into observable keys. Protect canonicalized or keyed identifiers, their mapping, and the cache store.
A cached answer needs more than text and an expiry timestamp. Store the conditions under which the answer was produced:
The envelope lets an operator answer: “Which users could receive this?”, “What evidence made it valid?”, “Which deployment produced it?”, and “What event must revoke it?” That complements data-quality observability: a source can arrive on time while a cached answer still points to the previous meaning.
A single arbitrary time-to-live hides the real invalidation problem. Compute answer freshness from the shortest-lived dependency and from explicit events.
For an answer based on three sources, a policy version, and an entitlement check, its safe lifetime cannot exceed the earliest expiry among them. It may need to end sooner when any of these events occurs:
Use versioned dependencies and reverse indexes so one event can locate affected entries. If invalidation requires scanning opaque answer text, the design will fail when speed matters. “Delete everything” is a useful emergency control, not a substitute for lineage.
Stale-while-revalidate may be reasonable for low-risk public guidance with a visible date. It is usually inappropriate for authorization, balances, approvals, incident status, or policy that can make an action permissible or forbidden.
Embedding similarity should retrieve candidates, not grant hits. A semantic candidate may be served only after deterministic invariants match:
The last test can combine rules, a small classifier, and a verifier, but the verifier is another fallible component. It must be evaluated on hard negatives: sentences that look similar while changing negation, actor, date, currency, threshold, jurisdiction, or requested action. If the verifier is uncertain, miss the cache.
Never let a generated confidence statement authorize reuse. Prefer deterministic exclusions and independent labelled tests. High similarity is useful for search; it is not an approval boundary.
Consider an assistant that explains procurement policy in Persian and English. A public rule says quotations are normally required above a threshold. A cached Persian answer was created for a Tehran operating company under policy PROC-17.4, using toman and a specific exception for emergency repairs.
The next query asks in Persian, “Can we buy this urgent replacement without three quotations?” It is semantically close. Before reuse, the gate discovers that the requester belongs to another subsidiary governed by PROC-19.1, the amount is above that subsidiary’s threshold, and the emergency exception requires a named approver. The semantic cache must miss.
The system can still save work:
If the query is only “What does three-way matching mean?” and the definition is public, versioned, non-personal, and identical across subsidiaries, a semantic candidate may pass. The difference is not linguistic similarity. It is whether every decision-changing variable remains equivalent.
A response cache is a content store and a routing system. Threat-model both.
Apply least privilege, encryption, namespace isolation, retention limits, deletion propagation, integrity protection, and access logging. Do not treat embeddings as anonymous; they are derived from source content and can still be sensitive. A system-prompt instruction not to reveal secrets is not a cache-isolation mechanism, which is consistent with OWASP’s warning that prompt restrictions can be bypassed.
Run semantic caching in shadow mode first. For each candidate hit, generate a fresh answer through the normal path and compare both against current evidence. Build a labelled set with ordinary pairs, exact duplicates, and adversarial near-matches.
Measure:
Overall hit rate is a vanity metric if unsafe traffic counts as success. Slice by language, tenant, workflow, policy, and rare entities; evaluate Persian, English, and cross-language retrieval rather than assuming parity.
Record each reuse decision in an audit-ready evidence trail: candidate entry, matched invariants, freshness result, policy version, served or rejected outcome, and later contradiction. Do not log unnecessary protected content merely to make the cache observable.
Enable provider prefix caching when its isolation, retention, and data handling fit the workload. Enable exact response caching only with a decision-complete key and event-driven invalidation. Limit semantic reuse to a low-consequence allow-list after shadow, hard-negative, privacy, deletion, and rollback tests pass.
Keep a global bypass and namespace-level purge. During a policy migration or incident, prefer a clean miss over a fast wrong answer. Revisit the gate when providers change cache semantics, the release passport changes, a new data class enters prompts, authorization becomes finer-grained, sources gain shorter validity, or users begin acting directly on answers.
The fastest response is not the one that arrives first. It is the one that remains correct after the user, policy, evidence, and system state are taken into account. Cache computation freely when its boundary is understood. Reuse conclusions only when equivalence is proved strongly enough for the consequence.
A practical control architecture for separating independent evidence, AI-influenced decisions, synthetic content, and production feedback before the next model learns from them.
Read More
A practical method for tracing personal data through AI pipelines, choosing deletion, rebuild, retraining, or unlearning, and proving that derived artifacts stay clean.
Read More
A practical guide to inventorying models, data, prompts, tools, evidence, and provenance so an exact AI release can be assessed, promoted, and rolled back.
Read MoreGet in touch with our team to discuss how we can help your business.