Structured outputs: validating AI responses before users see them

Free-text AI output is fine when a person reads it. It becomes a problem the moment code has to act on it. A dashboard needs a list of three priorities, not a paragraph that sometimes contains three priorities. A workflow needs a category from a fixed set, not a creative synonym. A database needs a valid identifier, not a product name the model half-remembers.

Structured outputs solve this by giving each AI response a contract: a JSON schema the output must match. Schema validation enforces the contract before anything reaches the application. At Oryvelon, validation is one of the standard jobs of our shared AI gateway: application code in every company receives either a valid object or a clear failure, never something in between. This article explains how we design schemas, what we check beyond the schema, and how retries and fallbacks fit together.

Why free text breaks products

Teams often start by asking a model for "a JSON object with these fields" in the prompt and parsing whatever comes back. It works in the demo. In production, the failure modes pile up:

• the model wraps the JSON in an explanation, or in a code block;
• a required field is missing on some inputs;
• a number arrives as a string, or with a currency symbol;
• a category comes back as a near-synonym of an allowed value;
• an array has five items when the screen has room for three;
• a field refers to a product, rule or record that does not exist.

Each of these is individually easy to handle. Together they spread defensive parsing code across the application, and every screen ends up with its own slightly different idea of what a valid response looks like. Worse, the last failure — an invented reference — often parses perfectly. It is valid JSON pointing at something that is not real.

Validation in one place fixes the first five. Reference checks fix the sixth.

Designing a schema the model can actually meet

Most model providers now support some form of constrained or schema-guided output, which makes format errors much rarer. It does not make schema design less important. A badly designed schema still produces badly designed data, just in valid JSON.

Our rules for a feature's schema:

Keep it small. Ask for the fields the application needs and nothing else. Every extra field is one more thing the model can get wrong and one more thing to test.

Close it. Disallow additional properties. If the model adds a field nobody asked for, that is a sign it is doing something unexpected, and the response should fail.

Prefer enums to free text. If a message can be one of six categories, the schema lists the six. "Refund request" and "refund_request" and "Refunds" are three different strings to code.

Use identifiers, not names. When the output refers to something in the input — a product, an order, a rule — it cites the identifier the input supplied. Names are ambiguous and easy to paraphrase. Identifiers can be checked.

Bound everything. Arrays have a maximum length. Strings have a maximum length. Numbers have ranges where ranges make sense.

Mark what is required. Optional fields invite the model to skip the hard ones.

Separate data from prose. If a response needs both a figure and a sentence about it, the figure is its own field, copied from the input, and the sentence is another. Code uses the figure. Users read the sentence.

Here is the shape of a briefing priority in plain terms: an object with a signal_id that must match one of the signals in the input, a severity from a three-value enum, a headline of at most 90 characters, and an explanation of at most 300. No other properties allowed. At most three priorities per briefing.

Validation is more than a schema check

A schema tells you the output has the right shape. It cannot tell you whether the contents are true. We run validation in three layers.

Layer 1: structure. Does the response parse, and does it match the schema — types, required fields, enums, lengths, no extra properties?

Layer 2: references. Does every identifier in the output exist in the input? If the model cites signal_id: s_14 and the input only contained signals s_1 to s_9, the response is rejected. This is the single most effective check against hallucination we have, because it turns a vague problem ("the model made something up") into a precise, mechanical one ("this ID was not supplied").

Layer 3: business rules. Feature-specific checks that the schema cannot express. Any figure quoted in a text field must match a figure in the input. A severity of "high" requires a signal flagged as high by the product's own logic. An explanation for KeşifAtlası must not state an eligibility outcome different from the one the rules engine produced.

Layer 3 checks are written by the product team, because only they know what "correct" means for their product. The gateway just runs them in a consistent place and records the results.

Rejecting hallucinated references in practice

The reference check deserves a closer look, because it shapes how we design the whole feature.

The pattern is simple. Before calling the model, the application assembles the facts the model is allowed to talk about, each with an identifier. After the call, every identifier in the output is checked against that set. Anything outside it fails.

For this to work, three design choices have to be made up front:

1. The model never looks things up. It receives facts; it does not fetch them. If the model can query a catalogue or a database during generation, the set of valid identifiers is no longer known in advance.
2. The input is complete for the task. If a briefing should be able to mention any product with low stock, then every low-stock product must be in the input. A model given a partial list will sometimes try to fill the gap.
3. Identifiers are opaque. An ID like p_2931 is harder for a model to invent plausibly than a name like "Blue Canvas Tote", and far easier to check.

This is the practical mechanism behind our principle that AI explains, verified data decides. The verified systems decide which facts exist. The model can only point at them. Retrieval-based features use the same idea: retrieved passages carry identifiers, and the output must cite passages that were actually retrieved for this request and this product. See Retrieval without leakage.

Retries: bounded and deliberate

When a response fails validation, the obvious move is to try again. Retries are useful, but unbounded retries are one of the most common ways AI features become slow and expensive without anyone noticing.

Our defaults:

• One retry for most features. Validation failures on a well-designed prompt are usually random, and a second attempt often succeeds.
• The retry may include the error. For structural failures, telling the model which field was missing can help. For reference failures, we usually retry without extra hints, because "you cited an ID that does not exist" sometimes leads the model to swap in a different wrong one.
• Retries count against the feature's budget and latency limit. If the first attempt already used most of the time allowance, the retry is skipped and the fallback runs.
• Every retry is logged with its reason, so a rising retry rate is visible on the product's weekly health check.

A feature where retries are frequent has a design problem, not a retry problem. It usually means the schema is too complicated, the prompt is ambiguous, the input is incomplete or the model tier is too small. The fix belongs in the prompt registry or the routing rule, not in more retries. See Prompt versioning and evaluation for production AI and Model routing: matching tasks to the right AI model.

What the application does when validation fails

After the retry, if the response still fails, the gateway returns a clear failure signal. The application then follows the feature's fallback, decided in advance.

The fallback depends on the feature:

What every fallback has in common is that the invalid output is never shown. A half-valid response is more dangerous than no response, because it looks authoritative. We describe the broader approach in Fallbacks and degraded modes for AI features.

A worked example: a refund spike at MerchNivo

MerchNivo is the AI e-commerce employee for Shopify stores. Imagine a store where one product suddenly receives several refund requests in a single day.

MerchNivo's own code detects the spike from the store's data. It creates a signal with an identifier, the product's ID, the refund count, the usual daily range and the refund reasons customers gave. That signal goes into the input for the daily briefing, alongside the other signals for the day. See Operations signals for Shopify stores.

The model returns a briefing. Validation runs:

• Structure. Three priorities, each with the required fields, severities from the enum, lengths within limits. Pass.
• References. The first priority cites the refund signal. The second cites a low-stock signal. The third cites a product ID that was not in the input — the model has linked the refund spike to a "similar product" it seems to have invented. Fail.

The gateway retries once without hints. The second response cites only supplied identifiers. Layer 3 then checks that the refund count mentioned in the explanation text matches the count in the signal. It does. The briefing is delivered.

If the retry had also failed, the merchant would have seen the same signals in a plain list — the refund spike, the low-stock warning — without the narrative. The facts would still arrive. Only the prose would be missing.

The merchant never sees the invented product. That is the whole point.

Structured outputs across our companies

The same mechanism serves very different products.

KeşifAtlası requires every explanation to cite rule identifiers from the verified rules database, and a Layer 3 check confirms the explanation does not contradict the rules engine's outcome. An explanation that sounds right but cites no rules is rejected.

CastLyra uses AI to help structure talent profiles and brand briefs. Its schemas keep contact details out entirely — there is no field for them — and a Layer 3 check rejects generated profile text that mentions credentials, agencies or past clients not present in the verified profile data. See Verification in talent marketplaces.

EduRelia uses schemas to keep learning content within the lesson's scope: a question object references a curriculum item from the input, answer options come from a bounded list, and explanations are length-limited and checked for age-appropriate language.

ZodiVela uses structure mostly for safety and consistency. A reading has defined sections with length limits, and post-response checks keep the language within its framing of reflection and entertainment rather than prediction or advice. See Positioning consumer AI as entertainment.

Monitoring validation over time

Validation produces some of the most useful operational data an AI product has. Every failure is labelled: which layer, which field, which reference, which prompt version, which model.

We watch three things:

• Validation failure rate per feature. A sudden rise usually means something changed: a prompt version, an input format, a model update on the provider side.
• Failure mix. Structure failures point at schema or prompt problems. Reference failures point at incomplete inputs or a model tier that is too small. Business-rule failures point at prompt instructions that need tightening.
• Fallback rate. How often users actually see the degraded experience. That is the number that matters to them.

Failures with personal information removed become cases in the feature's evaluation set, so the next prompt version is tested against exactly the inputs that tripped up the last one.

Changing a schema without breaking the product

Schemas change. A screen gains a field, a category is split in two, a limit turns out to be too tight. Because the schema is the contract between the model and the application, a careless change breaks both sides at once.

We version schemas alongside prompts. A prompt version in the registry names the schema version it was written for, and the application declares which schema versions it can read. A change then follows a predictable order:

1. Additive changes first. A new optional field is added to the schema, and the application learns to read it while still working without it.
2. The prompt follows. A new prompt version starts producing the field, and it is evaluated against the feature's test set like any other release.
3. Tighten last. Only once the new field is reliably present does it become required, in a further schema version.

Removing or renaming a field runs the same sequence in reverse. Splitting an enum value — say, "delivery issue" becoming "late delivery" and "damaged in transit" — is treated as a breaking change, because every downstream report that counted the old value needs to know.

The rule underneath is the same as for prompts: a published schema version is never edited. Logs record which schema version each response was validated against, so an old failure can always be understood in its own terms.

A checklist for a new structured feature

Before an AI feature that returns data goes live in one of our companies, the team can answer these:

• What exactly does the application do with each field? If nothing, remove it.
• Is every category an enum, every array bounded, every string limited?
• Which fields refer to input items, and do they use identifiers?
• Does the input contain every item the model might legitimately need to cite?
• What are the business-rule checks, and who wrote them?
• How many retries, and what is the latency limit including them?
• What does the user see if validation fails twice?
• Are failures logged with the layer and field, without storing raw personal data?

If any answer is unclear, the feature is not ready.

Common mistakes

• Trusting "valid JSON" as "valid output". Shape is not truth. Check references and business rules.
• Open schemas. Allowing extra properties lets unexpected behaviour through silently.
• Free-text categories. Every synonym becomes a bug somewhere downstream.
• Letting the model look things up. It makes the set of valid references unknowable in advance.
• Retrying until it works. It hides design problems and inflates cost and latency.
• Showing partially valid output. If it failed, it does not ship.
• Validation spread across the application. One place, one set of rules, consistent logs.

Summary

Structured outputs give every AI response a contract, and validation enforces it before anything reaches users. We keep schemas small, closed and built on enums and identifiers; check structure, references and business rules in three layers; reject any output that cites something not supplied in the input; allow one deliberate retry; and fall back to a non-AI view of the verified data when validation still fails. Running this in a shared gateway gives every Oryvelon company consistent protection against malformed and hallucinated responses, and turns every failure into a test case that makes the next version better.

Questions and answers

What are structured outputs in AI?

Structured outputs are AI responses returned in a defined machine-readable format, usually JSON matching a schema, so application code can use them directly instead of parsing free text.

How do you stop an AI model from inventing references?

Give the model only identifiers that exist in its input, require it to cite those identifiers in its output, and reject any response containing an identifier that was not supplied.

What should happen when an AI response fails validation?

Retry once within a bounded budget, then fall back to a simpler model, a non-AI version of the feature or a clear message to the user; never show the invalid output.

Published by Oryvelon, a digital company builder · About the group · Our companies · More in AI Operations

NextEarning user trust in AI products during onboarding →