Insights / AI Operations · · Updated · 8 min read

Latency and caching for AI features: keys, batching and streaming

Measure end-to-end AI latency, distinguish prompt caching from stored responses, and review access, freshness, batching and streaming before optimising.

For an AI feature, useful speed is the time from a user’s action to a result they can rely on. A quick first token is not enough if the answer is stale, belongs to another tenant or still needs validation. Measure the whole path before choosing an optimisation.

This guide proposes an engineering review for teams building digital products. It is not a description of deployed Oryvelon company features, a shared gateway or measured performance. Start with the task, its access boundary and its deadline; then compare the options below.

Where latency actually comes from

Break the request into observable stages instead of timing only the model call:

  • Data preparation: authorisation, source queries and retrieval.
  • Prompt construction: instructions, context and tool definitions.
  • Waiting: application queues, network and provider admission.
  • Generation: time to first output and time to completion.
  • Validation and delivery: parsing, policy checks, retries and rendering.

Record overlapping work separately from sequential work. A faster model cannot remove a slow source query, and moving work into a queue changes where the waiting happens rather than eliminating it.

Measure what the user feels

Define the first useful result for the feature: a validated summary, an accepted form result or a visible paragraph. Track that point as well as full completion. Use representative requests and inspect slow percentiles alongside failures; an average alone can hide a difficult subset.

Set a proposed end-to-end deadline that includes permitted retries. Decide what happens when it is exceeded: cancel, offer a bounded retry or return a clearly labelled limited result. Review fallbacks and degraded modes. A spinner should not silently outlive the product’s deadline.

Check the request path before adding a cache

The OpenAI latency guide discusses output length, request count, parallel work and model choice. It also cautions that shortening input can have a modest latency effect outside very large contexts. Treat such guidance as a hypothesis to measure on the actual workload, not a promised speedup.

  • Remove model work a reliable lookup or deterministic calculation can perform.
  • Reduce unnecessary output while checking that the answer still meets the task.
  • Run independent authorised source reads concurrently; retain dependency order where one result needs another.
  • Evaluate a smaller model against the same quality checks; see model routing.
  • Investigate repeated validation failures before adding automatic retries; see structured output validation.

Distinguish prompt caching from response caching

Provider prompt caching reuses eligible prompt processing. OpenAI’s prompt caching documentation describes matching rendered prefixes and model-specific cache boundaries and settings. It generates a new response; it does not promise the same answer for identical requests. Check the actual model’s requirements rather than assuming stable text always produces a cache hit.

Application response caching stores an answer or other result and serves that stored object later. Decide whether that reuse is appropriate: are the facts, permissions, output policy and intended variation still valid? Identify which layer owns each cache, its contents and its lifecycle.

Neither a prompt-cache hit nor a response-cache key proves that a caller may access data. Keep tenant authorisation explicit and review provider data handling separately.

Build a key from scope and relevant versions

For a proposed application response cache, document which inputs can change the valid answer. Typical review fields include:

  • Product and feature: prevent accidental collisions between different uses.
  • Authorised scope: tenant, user or permission-relevant context where the result is private. Resolve that scope from trusted membership checks.
  • Prompt and model configuration: version the instructions and relevant generation settings.
  • Source versions: include the records or corpus revision that determine freshness.
  • Input and output contract: include relevant facts, language and schema version.

Hashing a serialised input is an indexing technique, not encryption or an access check. Normalise only differences that are genuinely irrelevant; changing whitespace or field order can matter in some tasks. Test representative collisions and scope changes. See prompt versioning and evaluation.

A changed key prevents a lookup from matching an old entry only if every relevant input is captured. It does not delete the old object. Keep an explicit expiry, invalidation and deletion process.

Plan freshness, failures and invalidation

The AWS discussion of caching trade-offs considers expiry against tolerance for stale data, error handling and load when cached entries are missing. Its examples show why a cache needs operating and failure policies, not just a hit-rate target.

  • Stale facts or permissions: test source changes, revoked access and deleted records. An unexpired entry can still be incorrect or unauthorised.
  • Private copies: include retained answers in the data lifecycle, with access and removal rules.
  • Errors and limited results: distinguish them from successful answers. A deliberate short negative cache can reduce repeated failed work, but must not masquerade as a valid result.
  • Similar inputs: a semantic match is not evidence of equivalent facts or permissions. Evaluate the risk before reusing an answer.
  • Many simultaneous misses: consider bounded request coalescing within the correct scope, and test behaviour when the cache is unavailable.

Precompute work with a known freshness rule

A hypothetical scheduled report can be generated before a reader opens it if its source cutoff and acceptable age are defined. Display when its facts were current and recheck access when delivering the stored result. A later source correction may require invalidation or regeneration.

Precomputation also creates unused outputs and retained copies. Compare that cost with on-demand work, and define what happens if generation fails. A scheduled job is not evidence that a report will always be ready.

Batch work that can meet an asynchronous deadline

A queue, a provider batch interface and precomputation are related but different choices. A queue controls admission; a batch interface submits work for asynchronous processing; precomputation changes when results are prepared.

For example, the OpenAI Batch API provides an asynchronous workflow for groups of requests. Check its supported operations, completion window, status and error handling before choosing it. Do not treat a submitted batch as a completed result.

For a hypothetical catalogue-draft job, keep item identifiers, record partial failures and retry only the appropriate failed items. Ensure processing still meets the required review deadline. Interactive requests need a separate decision about waiting and cancellation.

Stream only what the interface may expose

Streaming can let a reader see useful text before the response finishes. It does not guarantee a particular first-token time or remove generation and validation work. Measure time to usable output, not just time to any bytes.

For a structured result, incremental parsing may be useful internally, but withhold actions or definitive values until the required object and checks are complete. If content needs whole-response review, hold it until that review succeeds. Chunk-level checks have different coverage and must not be presented as equivalent.

Define an interrupted-stream state. Show whether the response is incomplete, avoid saving a partial answer as a successful cache entry, and give the user an appropriate next step.

A hypothetical versioned rule explanation

Consider a service that explains a verified rule. This is a design example, not a description of KeşifAtlası’s implementation or measured results.

  • Load the authorised current rule and its version. Keep any eligibility decision separate from generated explanatory prose.
  • Decide whether the explanation is genuinely reusable. If it includes personal circumstances, it needs a separate scope and lifecycle.
  • Look up a result using the source version, prompt/configuration version, language and intended scope. Authorise the caller before returning it.
  • On a miss, generate and validate the explanation. Store only an eligible successful result with its provenance and expiry.
  • When the rule changes, invalidate or version the result; test permission changes and deletion separately.

Compare cold and warm requests, correctness and failure behaviour. A hoped-for hit rate or an attractive diagram is not a performance result. See rules engines and eligibility.

Review rate limits, queues and fairness

Define admission and concurrency limits for the actual service. Separate interactive and background work where needed, and measure queue age, rejection and cancellation as well as throughput. Consider how one tenant’s workload affects another.

Use bounded retries and observe provider retry guidance. Separate project credentials do not by themselves prove independent quotas or capacity; verify the relevant account, project and model limits. Review any shared gateway as a design component rather than assuming it removes these dependencies.

Choose optimisation work from observed need

Compare the operating cost and complexity of a change with its observed benefit. Running multiple speculative generations can spend work that will be discarded, while an added cache creates another service to monitor. Measure total request cost and quality rather than promising a fixed saving.

Start with a representative baseline and change one well-defined part of the path. Keep the change only when the result meets the feature’s quality, privacy and reliability requirements. See AI product cost discipline.

A review checklist

  • Define the useful-output milestone and end-to-end deadline.
  • Separate provider prompt caching from application result reuse.
  • Verify authorised scope, source versions and relevant key inputs.
  • Test stale data, revoked access, deletion and cache unavailability.
  • Define retries, partial results, stream interruptions and asynchronous completion.
  • Compare representative cold/warm latency, quality, failures and operating cost.

Record the decision and observed results in the feature’s source-of-truth document. Review again when permissions, providers or input/output contracts change.

Summary

Measure the entire path to a useful result, then choose an appropriate change. Prompt reuse and stored answers are different mechanisms. Safe response reuse requires current facts, explicit authorisation and a lifecycle for retained objects. Batching and streaming should fit the task’s deadline and validation needs. Verify the result on the actual workload instead of turning a general technique into a company performance claim.

Questions and answers

Is prompt caching the same as caching an AI response?

No. Provider prompt caching reuses eligible prompt processing while generating a new response. An application response cache serves stored output and needs its own authorisation, freshness and retention checks.

Does a tenant ID in the cache key guarantee isolation?

No. Resolve the scope from trusted membership checks, authorise access and test the full data path. Relevant permissions and source versions must be covered; expiry and a hash do not replace those controls.

When should an AI feature stream its answer?

Use streaming when partial text is useful and permitted to reach the reader. Withhold results that need complete validation, and define what happens when the stream stops or a check fails.

NextRAG without data leakage: per-product and per-tenant retrieval →