What an AI gateway does — and what it should never do

When several products use AI, a question arrives quickly: should each product call model providers directly, or should calls go through something shared?

Calling directly is simplest at first. Each team picks a model, adds an API key and writes some code. But after a few products, the same problems appear everywhere: no consistent budgets, no shared monitoring, no standard way to handle failures, keys scattered across projects, prompts edited in place without history, and no easy way to switch models when prices or quality change.

At Oryvelon, every AI feature goes through the Oryvelon AI Gateway: a thin shared layer that standardises how products use models. This note explains what it does, how it is designed, and the line it must never cross.

The gateway in one sentence

The AI gateway is a common, product-aware layer between our applications and model providers that applies each product's routing, budgets, limits, safety checks, logging and fallbacks — without ever mixing one product's prompts, knowledge or users with another's.

What the gateway is responsible for

Request policy

Each product has a policy: which model families it may use, which features may call which models, maximum input and output sizes, and rate limits per user and per product. The gateway enforces the policy on every request. A consumer feature cannot accidentally call an expensive reasoning model; an internal batch job cannot flood a provider.

Model routing

Simple tasks such as classification or short summaries go to smaller, cheaper models. Complex reasoning goes to stronger models, sparingly. The routing rules live in configuration, per product and per feature. See Model routing: matching tasks to the right AI model.

Prompt registry

Important prompts are registered with an identifier, a version, an owner and a test status. Applications reference prompts by identifier and version rather than embedding text in code. That makes changes visible, reversible and testable. See Prompt versioning and evaluation for production AI.

Budget controls

Every product has its own monthly budget with a soft limit (alert) and a hard limit (switch to degraded mode). Budgets can also be set per feature or per user where abuse is possible. See Cost discipline for AI products.

Safety hooks

Before a request leaves, the gateway can apply product-specific checks — for example, removing personal information that the model does not need. After a response returns, it can run output checks: content boundaries for consumer products, format checks for structured outputs. See Safety boundaries for consumer AI products.

Observability

For every request the gateway records latency, token usage, cost, model, prompt version, success or failure, and retries — per product. Logs are filtered so that sensitive raw content is not stored by default.

Schema validation

Where a product asks for structured output, the gateway validates the response against the expected schema before returning it. Invalid responses are retried within limits or reported as failures, so application code never receives malformed data. See Structured outputs.

Fallbacks

If the primary model fails — timeout, provider outage, repeated invalid output — the gateway follows the product's approved fallback: a secondary model, a simpler prompt, or a signal to the application to use its non-AI degraded mode. See Fallbacks and degraded modes for AI features.

Caching

For deterministic, low-variance requests, the gateway can return cached responses. Cache keys include the product, the prompt version and a fingerprint of the input, so one user's output is never served to another and cached answers expire when inputs change. See Latency, caching and batching.

What the gateway must never do

This part matters more than the list above. A shared AI layer is exactly the kind of component that can quietly break the boundaries between companies. So the gateway has explicit rules about what it must not do.

It never merges prompts. Each product's prompts are registered under that product. There is no "shared system prompt" that every product uses.

It never shares knowledge bases. If a product uses retrieval over its own documents, that index belongs to the product and is only queried on its behalf. See Retrieval without leakage.

It never builds user profiles. The gateway does not know, or try to know, that the same person uses two products. It sees requests from products, not people across products.

It never pools keys or budgets. Each product has its own AI project and API keys with the provider, and its own budget. The gateway routes a product's requests using that product's credentials.

It never becomes the source of truth. The gateway does not store business facts and does not make business decisions. Facts come from each product's verified systems; decisions come from each product's rules. See AI explains, verified data decides.

It never logs sensitive content by default. Operational metrics are recorded; raw inputs and outputs are only kept where a product explicitly needs them for evaluation, with personal information filtered and retention limited.

How it is designed

The gateway is intentionally thin. It is a small service plus a client library that each product uses. Most of its behaviour is configuration: policies, routes, prompt references, budgets and fallback chains, stored per product.

A typical request flows like this:

1. The product calls the client with a feature name, a prompt identifier and version, and the input data (which already contains the verified facts).
2. The gateway loads the product's policy for that feature and checks limits and budget.
3. Pre-request safety checks run.
4. The request is routed to the configured model using the product's credentials.
5. The response is validated, post-response checks run, and metrics are recorded.
6. On failure, the configured fallback runs.
7. The product receives either a valid response or a clear failure signal.

Built when needed, not before

It is easy to over-engineer something like this — to build a large internal AI platform with every feature imaginable before any product needs it. That is what our master plan calls "architecture theatre", and we deliberately avoid it.

The gateway grew in steps. Budgets and logging came first, because every product needed them from day one. Routing followed when products began to mix small and large models. The prompt registry arrived when prompts became numerous enough that editing them in code was risky. Caching was added for the products where it clearly paid off. Each capability exists because a real product needed it.

What it gives each product

For a product team, using the gateway means:

• cost limits and monitoring from the first request;
• a standard way to switch or combine models;
• versioned prompts and a place to run evaluations;
• consistent fallbacks when providers fail;
• safety checks appropriate to the product;
• no need to handle provider credentials in application code.

And it means all of that without giving up the product's independence. If a product were ever separated from the group, it could keep running the client with its own configuration, or replace it — its prompts, keys and data were always its own. See Designing every company so it could stand alone.

A worked example: one request, end to end

Abstract descriptions only go so far, so here is what happens when a merchant opens their morning briefing in MerchNivo.

The briefing feature asks the gateway for a short, structured summary of yesterday's store activity. Before the call is made, MerchNivo's own code has already pulled the numbers from the store: orders, revenue, refunds, low-stock products, abandoned checkouts. Those figures are the facts. The model's job is only to put them into plain language and suggest which two or three things deserve attention first.

The request reaches the gateway carrying four things: the product identifier (MerchNivo), the feature name (daily_briefing), a prompt reference (briefing.summary, version 7) and the input payload with the verified numbers.

The gateway looks up MerchNivo's policy for daily_briefing. The policy says this feature may use a mid-sized model, has an output limit suitable for a short summary, and must return JSON matching the briefing schema. It checks that MerchNivo's monthly budget has room and that this store has not exceeded its daily allowance of briefings.

A pre-request hook strips anything the model does not need — customer names and emails that happen to appear in order data, for example. The model gets "3 refunds today, two for the same product", not who asked for them.

The call goes out using MerchNivo's own credentials. The response comes back, and the gateway validates it against the schema: a headline, a list of up to three priorities, each with a reference to the underlying figure. One of the priorities refers to a product that was not in the input. That is a hallucinated reference, so the validator rejects the response and the gateway retries once with the same prompt. The second response is valid.

Metrics are recorded: model, prompt version, tokens, cost, latency, one retry, success. No raw content is stored. MerchNivo receives a clean, validated object and renders it in the merchant's dashboard.

If both attempts had failed, the gateway would have returned a clear failure, and MerchNivo would have shown its non-AI fallback: the same numbers in a simple table, without the narrative. The merchant still gets the facts. They just get them without the prose.

Configuration per company

Because the same gateway serves very different companies, most of what makes it useful is configuration rather than code. A few examples show how differently it is set up.

ZodiVela is a consumer product with a large number of short interactions. Its configuration emphasises per-user rate limits, strict content boundaries on output, small models for most tasks and aggressive budget alerts. Readings are framed for reflection and entertainment, and the post-response hooks check that the language stays within those limits.

KeşifAtlası uses AI to explain rules, never to decide them. Its configuration requires that every explanation references rule identifiers from its verified rules database, and the validator rejects responses that make eligibility claims without such a reference. Questions the system cannot answer confidently are routed to a person. See Rules engines for eligibility.

EduRelia handles content for students. Its configuration minimises what is sent — no names, no identifiers beyond what the task requires — and applies age-appropriate output checks. Nothing from EduRelia's requests is retained for evaluation without explicit, documented need. See Data minimisation for children in edtech.

CastLyra uses AI to help structure profiles and briefs. Its configuration keeps talent contact details out of prompts entirely and validates that generated profile text does not invent credentials.

Same gateway, four very different sets of rules. That is the point: the shared layer handles the mechanics, and each company decides what is appropriate for its own customers.

What we monitor

The gateway's metrics are only useful if someone looks at them. Each company's owner sees a small set of indicators for their own product, as part of the weekly health check described in The operating cadence behind a portfolio of digital businesses.

• Cost per useful outcome. Not cost per call, but cost per briefing delivered, per reading completed, per explanation shown. This is the number that tells us whether AI is paying for itself.
• Validation failure rate. A rising rate usually means a prompt has drifted, an input format has changed or a model update has altered behaviour.
• Fallback rate. How often users receive the degraded experience. A small number is healthy; a growing one needs attention.
• Latency at the slow end. Averages hide the experiences that frustrate people. We watch the slowest requests.
• Budget position. Where each product stands against its soft and hard limits for the month.

Alerts fire on sudden changes in any of these. Slow trends are discussed in the monthly snapshot.

Evolving the gateway without breaking products

Shared infrastructure has a particular risk: a change made for one product can surprise another. We handle gateway changes with the same discipline we apply to shared code in general.

The client library is versioned, and each product upgrades on its own schedule. New capabilities are added behind configuration flags that default to off, so a product only gets new behaviour when it asks for it. Changes to default behaviour — for example, a stricter logging filter — are announced to every product owner before they take effect, with a date and a way to test in advance.

Model changes are handled per product, not globally. When a provider releases a new model, one product evaluates it against its own test set first. If it performs better, that product switches. Others follow when their own evaluations agree. No product is moved to a new model because another product preferred it.

Build, buy or skip: the trade-offs

A gateway is not the right answer for every team. The choice depends on how many products use AI and how different they are.

Off-the-shelf gateway software and provider-side features can cover a lot of the mechanics: routing, rate limits, usage tracking. What they cannot decide for you is the policy — which product may send what, what must be filtered, what counts as a valid answer. That part is always yours. Our own layer is mostly that policy, expressed as configuration.

The wrapper in the first row matters more than it looks. Even a single product benefits from having every model call go through one function in its own codebase. When a second product appears, that function is where the gateway client slots in.

Common mistakes when building an AI gateway

Putting business logic in the gateway. It starts small: a formatting rule here, a pricing check there. Soon the gateway knows things about each product that only the product should know. Business rules belong in the product.

One shared system prompt. It is tempting to add a group-wide preamble to every request "for consistency". It mixes products' voices and boundaries, and a change for one product silently changes all of them.

Logging everything, just in case. Full request and response logs are convenient for debugging and a liability for everything else. Default to metrics; keep raw content only with a named purpose, filtering and a retention limit.

A single budget for all products. It hides which product is expensive and lets one product's spike starve the others. Budgets are per product, and often per feature.

Treating the gateway as a proxy only. A gateway that forwards requests without validating responses moves the hardest part — deciding whether an answer is usable — back into every application.

Global model switches. Moving every product to a new model at once because it looked better in one product's tests. Each product evaluates for itself.

A readiness checklist for a new AI feature

Before any Oryvelon product ships a feature that calls a model, its configuration answers these questions:

• Which prompt identifier and version does it use, and who owns it?
• Which models may it call, with which limits on input and output size?
• What schema must the response match, and what happens when it does not?
• Which facts in the input come from verified systems, and which fields are removed before the call?
• What are the per-user and per-feature limits?
• What is the fallback, and what does the user see in degraded mode?
• What is logged, what is not, and for how long?
• Which metric tells us the feature is worth its cost?

If any answer is missing, the feature is not ready, however good the demo looks. For a store-facing feature in Noveniq or a reading in ZodiVela, the answers differ completely — but the questions are the same.

Where keys and environments fit

Credentials deserve a short note of their own. Each product's provider keys live in a secrets store, scoped to that product and environment. Application code never sees them; the gateway retrieves the right key for the calling product at request time.

Development and preview environments use separate keys with small budgets, and they never receive production customer data as prompt input. Test sets for evaluation are built from synthetic or properly anonymised examples. A mistake in a preview build should cost a few cents and expose nothing.

Summary

An AI gateway is a useful piece of shared infrastructure when it standardises how models are called and stays out of what products say to them. Ours handles routing, budgets, prompts, safety, observability, validation, fallbacks and caching for every Oryvelon product — and draws a hard line at merging prompts, knowledge, keys or users. Facts stay with each product's verified systems. Decisions stay with each product's rules. The gateway just makes calling a model safer, cheaper and easier to change.

Questions and answers

What is an AI gateway?

A shared layer that sits between applications and AI model providers, handling routing, budgets, rate limits, logging, validation, fallbacks and safety checks in a consistent way.

Does the Oryvelon AI gateway share data between companies?

No. Each product has its own AI project, keys, prompts, knowledge base, budget and logs. The gateway standardises how models are called, not what is sent.

Why use a gateway instead of calling models directly?

It gives every product cost limits, monitoring, fallbacks and safety hooks without each team rebuilding them, and makes switching models a configuration change.

Published by Oryvelon, a digital company builder · About the group · Our companies · More in AI Operations

NextModel routing: matching tasks to the right AI model →