Latency and caching for AI features: keys, batching and streaming

Speed is a feature. A good AI answer that takes fifteen seconds loses to a decent one that takes two, and both lose to a plain table that appears instantly. Cost follows the same shape: the fastest request is often the one you never had to send.

This note covers how we approach latency and caching for AI across Oryvelon's companies: where we measure, what we cache and how we build cache keys, when we batch, and when streaming is worth it. The mechanics sit in our shared AI gateway; the decisions sit with each product.

Where latency actually comes from

When someone says "the AI is slow", the model is usually only part of the story. A typical AI feature spends time in several places:

1. Gathering facts. Querying the store, the rules database or the product's own tables to build the input.
2. Building the prompt. Loading the registered prompt, adding the facts, applying pre-request safety hooks.
3. Waiting in line. Rate limits, provider queues, network.
4. Time to first token. How long the model takes to start responding. This grows with input length and model size.
5. Generation. How long it takes to produce the full output. This grows with output length.
6. Validation and post-processing. Schema checks, post-response hooks, possibly a retry.
7. Rendering. Getting the result on screen.

Most teams optimise step 5 and ignore the rest. In our experience the biggest gains usually come from steps 1, 4 and 6: fetching facts in parallel, sending less input, and avoiding retries by writing prompts that pass validation the first time.

Measure what the user feels

We measure latency from the moment the user acts to the moment they see something useful. The gateway also records the model call on its own, but the end-to-end number is the one that matters.

And we look at the slow end, not the average. An average of three seconds can hide a tenth of requests that take twelve. Those are the ones people remember and complain about. The gateway reports slow-end latency per product and per feature, and it is one of the indicators each company owner sees in their weekly product review.

Every interactive feature also has a latency budget written into its configuration: the total time it may take, including any retries and fallbacks. When the budget runs out, the feature moves to its degraded mode rather than keeping the user waiting. See AI fallbacks and degraded modes.

The cheapest speedups

Before caching or streaming, there are a few changes that almost always help.

Send less. Input length drives both cost and time to first token. A MerchNivo briefing does not need every order from yesterday; it needs the aggregated figures and the few outliers worth mentioning. Summarise in code, then ask the model to explain the summary.

Ask for less. Output length drives generation time. A cap on output tokens, plus a prompt that asks for three priorities rather than "a thorough analysis", often halves latency with no loss of usefulness.

Use a smaller model for small jobs. Classification, extraction and short rewrites rarely need a large model. See Model routing.

Fetch facts in parallel. If a feature needs data from three sources, fetch them at the same time. This is ordinary engineering and it is often the biggest single win.

Get it right the first time. Every retry after a validation failure roughly doubles latency for that request. Tightening a prompt so it passes validation reliably is a latency fix. See Structured outputs and schema validation.

Caching: what is safe to cache

Caching model responses is powerful and easy to get wrong. The failure that worries us most is not a stale answer. It is the wrong person's answer.

So we are conservative about what is eligible. A request can be cached only if:

• the output depends only on the input sent, not on hidden state, time of day or randomness the product relies on;
• the feature is low-variance by design — the same input should produce essentially the same useful output;
• the product has explicitly enabled caching for that feature in its configuration.

Good candidates in our companies include explaining a specific rule in KeşifAtlası's rules database, generating a standard description for a product category in Noveniq's catalogue, or classifying the type of a support message. Poor candidates include anything personal, anything that combines one user's data with a question, and anything creative where variety is part of the value.

Building a safe cache key

The cache key is where safety lives. Ours is built from these parts:

Two parts deserve a closer look.

The prompt version. Including it means cache invalidation for prompt changes is automatic. When a new version of rule.explain goes live, every request produces a new key, and the old entries simply age out. Nobody needs to remember to clear anything. This is one of the reasons we register prompts with versions in the first place; see Prompt versioning and evaluation.

The input fingerprint. We normalise the input before hashing — consistent ordering of fields, trimmed whitespace, stable number formatting — so trivially different inputs do not miss the cache. But we fingerprint the facts as well as the question. If KeşifAtlası updates a rule's text, the input changes, the fingerprint changes, and the cached explanation of the old text is no longer returned. The cache follows the truth instead of lagging behind it.

We also set an expiry on every entry, even when the key should make staleness impossible. Keys can be designed imperfectly, and an expiry is a cheap safety net.

Where caching goes wrong

A few mistakes we design against:

Caching personal data. If the input contains anything about a specific person, the cached entry is effectively a copy of personal data with its own retention period. We avoid it, and where it is truly needed, the entry is scoped to that user, short-lived and covered by the product's retention rules. See Data retention and deletion.

Caching degraded output. If a fallback response is stored under the same key as a full response, users can keep receiving the fallback after the model recovers. Degraded output is either not cached or cached under a separate, short-lived key.

Caching failures. A validation failure or safety block should never be cached as an answer.

Semantic caching without care. Matching "similar" questions to cached answers is appealing, but similarity is not sameness. For a rules product, two questions that look alike can have different correct answers because one detail differs. We only use exact-match caching for anything where correctness matters.

Sharing a cache across products for efficiency. Even if two products happened to ask identical questions, we would not share the entry. The saving is tiny; the precedent is bad. Shared infrastructure, separate data — see Shared infrastructure, separate data.

Precompute what nobody is waiting for

The fastest request is one that already happened. A surprising amount of AI work in real products does not need to run while a person waits.

MerchNivo's morning briefing is the clearest example. Merchants open it in the morning. The underlying numbers are final shortly after midnight in the store's time zone. So the briefing for each store is generated in the early hours, validated, and stored. When the merchant opens the dashboard, it loads instantly. If generation failed overnight, there is time to retry before anyone looks.

Catalogue content for Noveniq — draft product descriptions, alt text, attribute extraction — is produced in batches when products are added, then reviewed by a person before publishing. Nothing about it needs to be interactive.

Evaluations of new prompts and models run as scheduled jobs against stored test sets, never on live traffic.

Precomputing has its own cost: you pay for outputs some users will never look at. For a daily briefing that most merchants open, that is fine. For a detailed report that only a few people request, generating on demand is cheaper. We decide per feature, based on how often the output is actually used.

Batching

Batching means grouping many independent requests and sending them together, typically through a provider's batch interface or simply a queue that processes jobs steadily instead of all at once.

It helps in three ways. Batch interfaces are often cheaper per request. A queue smooths load, so a burst of work does not hit rate limits. And failures are easier to handle: a failed item goes back in the queue instead of producing an error on someone's screen.

The trade-off is time. Batched work may take minutes or hours to complete. That is fine for the overnight briefings and catalogue drafts above. It is not fine for anything interactive.

A simple rule we use: if a person is looking at a spinner, it is interactive. Otherwise, it is a candidate for the queue.

Streaming: helpful, but not everywhere

Streaming sends output to the user as it is generated instead of waiting for the full response. It makes long responses feel much faster, because the reader starts reading within a second or two.

It fits well where the output is prose and a person reads it top to bottom. A ZodiVela reading is a good example: the first paragraph appears quickly, and the reader is already engaged while the rest arrives.

It fits poorly in three situations.

Structured outputs. If the product needs a validated JSON object — MerchNivo's list of priorities with references to figures, KeşifAtlası's explanation with rule identifiers — it cannot show half of it safely. It needs the complete object, validated against the schema, before anything reaches the user.

Content that must pass post-response checks. For consumer products with content boundaries, streaming means text reaches the screen before the whole response has been checked. We handle this for ZodiVela by running checks on each completed paragraph before it is displayed, with the stream held briefly while a paragraph finishes. For EduRelia, where users are students, we generally do not stream tutoring responses at all; the full answer is checked before it is shown. A slightly slower answer is a fair price. See Safety boundaries for consumer AI products.

Short answers. If the whole output takes a second to generate, streaming adds complexity for no real benefit.

A worked example: KeşifAtlası's rule explanations

To show how these pieces combine, here is how a single feature is tuned.

When a user completes KeşifAtlası's free eligibility test, the rules engine produces an outcome and a list of rules that applied. For each rule, the user can see a plain-language explanation.

• Facts first. The rules engine runs in code and returns instantly. The outcome is shown immediately, before any model call.
• Cache per rule. The explanation of a rule depends only on the rule's current text and the prompt version, not on the user. So explanations are cached with a key of product, feature, prompt version, model and a fingerprint of the rule text. Most requests are cache hits.
• Precompute on change. When a rule is added or updated in the verified database, its explanation is regenerated in the background, validated and reviewed. The first user to see it never waits.
• Personal part separate. Where the explanation needs to mention the user's own answers — "because you indicated…" — that part is assembled from templates in code rather than generated, so nothing personal ever enters the cache.
• No streaming. Explanations are short and structured, with rule identifiers the validator checks.

The result is a feature that feels instant, costs very little per user, and updates itself when the rules change. See Rules engines for eligibility.

Rate limits, queues and fairness

Latency is not only about one request. When many requests arrive together, the product's own rate limits and the provider's limits decide who waits.

Each product has limits per user and per feature, enforced by the gateway. That protects the budget, but it also protects latency for everyone else: one heavy user running dozens of requests in a minute should not slow the experience for the rest. On CastLyra, for example, a brand drafting several briefs in quick succession gets queued gracefully rather than blocking talent who are editing their own profiles at the same time.

Interactive and background work also run in separate lanes. A large overnight batch for one product should never compete with a person waiting on a screen. Background jobs use their own queue with their own limits, and they back off when interactive traffic is high.

Finally, provider limits are per product, because each company has its own AI project and keys. A traffic spike in one company does not consume another company's allowance, which is one more quiet benefit of keeping products separate.

What we deliberately do not optimise

Not every millisecond is worth chasing. We skip optimisations that add complexity without a clear effect on users or cost.

We do not build custom model hosting to shave latency for small products. We do not add semantic caches, speculative generation or multi-model racing — sending the same request to two models and taking the fastest — unless a feature has a proven need. Racing, in particular, doubles cost to save a second, and it complicates evaluation because two models' outputs reach users.

We also avoid tuning features nobody uses. The weekly metrics show which features carry real traffic. Those get attention. A feature used a handful of times a week can take a few extra seconds; if it turns out to matter, the numbers will say so. It is the same cost discipline we apply everywhere else: spend effort where it changes an outcome.

A short checklist

For each AI feature, we answer these in the company's source-of-truth document:

1. What is the end-to-end latency budget, including fallbacks?
2. Can any part of the work be precomputed or batched?
3. Is the input as small as it can be? Is the output capped?
4. Is caching enabled, and does the key include product, scope, feature, prompt version, model and input fingerprint?
5. Does any cached content contain personal data? If yes, why, and for how long?
6. Is streaming used, and if so, how are post-response checks applied?
7. Which latency and cache hit indicators does the owner watch?

Summary

Latency and caching for AI are mostly about moving work to the right place. We measure the end-to-end delay users feel, especially at the slow end; shrink inputs and outputs and route small jobs to small models; precompute and batch anything nobody is waiting for; and cache only low-variance, non-personal requests using keys built from product, scope, feature, prompt version, model and an input fingerprint, so answers never cross users or products and refresh automatically when the prompt or the facts change. Streaming is used where people read long prose, not where outputs must be validated first.

Questions and answers

How do you cache responses from a language model safely?

Build the cache key from the product, the feature, the prompt version, the model and a hash of the exact input, and cache only low-variance requests whose output does not depend on anything outside that input. Set expiry times and never share cached entries between products or tenants.

When should an AI feature use streaming?

Streaming helps when the output is long prose that a person reads as it arrives. It is less useful, and sometimes harmful, for structured outputs that must be validated in full before they can be shown.

What is the easiest way to reduce AI latency?

Move work off the request path: precompute or batch anything the user is not actively waiting for, send smaller inputs, and route simple tasks to smaller models. These changes usually matter more than tuning the model call itself.

Published by Oryvelon, a digital company builder · About the group · Our companies · More in AI Operations

NextRAG without data leakage: per-product and per-tenant retrieval →