Model routing: matching tasks to the right AI model

Most AI features do not need the biggest model available. A support message needs a category. An order note needs three fields pulled out of it. A daily summary needs a few numbers turned into sentences. These are small jobs, and paying large-model prices for them is one of the quickest ways an AI product loses its margin.

Model routing is the discipline of matching each task to the right AI model: small, fast models for the routine work, larger models for the problems that actually need reasoning, and a clear rule for every feature saying which is which. At Oryvelon, routing is one of the first things our shared AI gateway does for every product. This article explains how we think about it, how we write the rules, and the mistakes we try to avoid.

Why one model for everything is the expensive default

When a team builds its first AI feature, it usually picks one capable model and uses it everywhere. That is a reasonable way to start. It removes a decision while the product is still finding its shape.

The problem is that the default hardens. Six months later the same large model is classifying tickets, generating subject lines, extracting addresses and writing long explanations. Most of those calls are paying for capacity they never use. Latency is higher than it needs to be. And because nobody ever measured a smaller model on the simple tasks, nobody knows how much is being wasted.

There is a quality argument too, and it surprises people. Larger models are not always better at narrow tasks. For a tight extraction job with a strict schema, a small model with a clear prompt can be just as accurate and more consistent, because there is less room for it to be creative. A model that likes to elaborate is a liability when you want exactly three fields and nothing else.

So the question is never "which model is best?" It is "which model is good enough for this specific task, at the lowest cost and latency?" That question has a different answer for every feature.

The task, not the product, is the unit of routing

A common mistake is to route by product: "ZodiVela uses model A, MerchNivo uses model B." Products are bundles of very different tasks. Routing has to happen one level down.

We group tasks into a handful of types, because each type has a predictable profile:

The tiers are starting points, not laws. Plenty of summarisation works well on small models; some classification tasks with subtle categories need more. But the table gives a team a sensible first guess, and it forces the right conversation: what kind of task is this, really?

Once a feature is described as a task type, the routing rule almost writes itself.

How we write a routing rule

Every AI feature in an Oryvelon company has a routing entry in that product's gateway configuration. It is short, and it always has the same parts:

• Feature name. The stable identifier the application uses, such as daily_briefing or ticket_triage.
• Task type. One of the types above.
• Primary model. The model family and tier the feature uses by default.
• Fallback. What happens if the primary fails: a second model, a simpler prompt, or a signal to use the non-AI degraded mode.
• Limits. Maximum input and output size for this feature.
• Owner. The person responsible for the rule.
• Reason. One or two sentences explaining why this model was chosen, with a reference to the evaluation that supports it.

The last field is the one teams skip, and it is the one that matters most a year later. "Mid-tier because the small model missed 1 in 8 refund-related priorities in the briefing test set" is a reason someone can re-check when a new model is released. "Seemed better" is not.

A rule might look like this in plain terms: ticket_triage is a classification task. Primary: small model. Fallback: a different small model, then rule-based keyword tagging. Max input: 2,000 tokens. Owner: product lead. Reason: small model matched the labelled test set at the agreed threshold; mid-tier gave no meaningful improvement.

That is all a routing rule needs to be. It is readable by anyone on the team, not only the engineer who wrote it.

Start small and earn your way up

Our default is to begin every feature on the smallest model that could plausibly do the job, and to move up only when evaluation proves a gap.

This is the opposite of how most teams work. They start large, get a good result, and never test whether they needed it. We prefer the opposite order for three reasons.

First, it forces a test set to exist. You cannot decide a small model is "not good enough" without examples of what good enough means. That test set is then reused for every future model decision, and it becomes part of the prompt's record in the registry described in Prompt versioning and evaluation for production AI.

Second, it makes the cost of quality visible. If the mid-tier model scores slightly higher and costs several times as much, that is now a choice someone has made on purpose, with numbers in front of them.

Third, it keeps the product honest about what the AI is for. When a feature seems to need a very large model, that is often a sign the task is badly shaped: the prompt is asking the model to find facts it should have been given, or to make a decision that should come from rules. Moving the facts and the decision out of the model frequently makes a small model sufficient. This is the practical side of our principle that AI explains, verified data decides.

Measuring the trade-off honestly

The cost side of routing is easy to see. The quality side takes more work, and it is where routing decisions usually go wrong.

We look at four numbers for each candidate model on a feature's test set:

1. Pass rate against the feature's quality bar. Defined per feature. For extraction, exact field matches. For summaries, a rubric that checks faithfulness to the input and coverage of the required points.
2. Validation failure rate. How often the output fails the schema or reference checks described in Structured outputs: validating AI responses before users see them.
3. Latency at the slow end. Not the average. The slowest tenth of requests is what users remember.
4. Cost per useful outcome. Total spend, including retries and fallbacks, divided by the number of outputs that were actually delivered to a user.

The fourth number changes decisions. Imagine a small model that costs a fraction of a mid-tier model per call but fails validation often enough to need retries, and occasionally falls through to the fallback. Its cost per delivered result can end up close to the mid-tier model's, with worse latency. On paper it was cheaper. In practice it was not.

The reverse happens too. A large model with a great pass rate may be only marginally better than a mid-tier one on the cases users notice. If the gap sits in edge cases that the product handles elsewhere — for example by escalating to a person — the extra spend buys very little. We cover the budget side of this more fully in Cost discipline for AI products.

Fixed rules first, dynamic routing later

There are two broad styles of routing.

Fixed routing sends every request for a feature to the same model. It is predictable, easy to audit and easy to reason about when something breaks. If a user reports a bad briefing, you know which model produced it.

Dynamic routing decides per request. A cheap classifier, or a simple rule, looks at the input and chooses a tier: short, familiar inputs go to the small model; long, unusual or ambiguous ones go to a larger model. Done well, it gives large-model quality on hard cases at small-model cost on the rest.

We start every feature on fixed routing. Dynamic routing earns its place only when three things are true:

• the feature's inputs vary clearly in difficulty;
• the difference can be detected cheaply and reliably before the main call;
• the test set shows the small model failing mostly on the hard inputs, not randomly.

If the small model's failures are scattered across easy and hard cases alike, a difficulty router will not help. You will just add a second model call and a new place for bugs.

When we do use dynamic routing, the router's rules are as explicit as possible. Input length, the presence of certain fields, the number of items to compare, a language flag. A second model deciding which model to call is sometimes justified, but it doubles the surface you have to evaluate, so we use it sparingly.

Escalation within a request

A related pattern is escalation on failure. The small model tries first. If its output fails validation, or a confidence check falls below a threshold, the request is retried on a larger model before anything is shown to the user.

This works well for tasks where the small model succeeds most of the time and failures are detectable. Extraction with a strict schema is the classic case: a missing required field or an invalid reference is easy to spot, and the retry is cheap relative to the savings on all the calls that succeeded first time.

It works badly where failure is not detectable. If a summary is fluent but subtly wrong, validation will pass it and the escalation will never trigger. For those tasks, escalation-on-failure gives a false sense of safety, and it is better to pick the right tier up front.

Escalation between models is not the same as escalation to a person. When the question itself is too hard or too consequential for any model, the right route is a human, and that is designed separately. See Designing human escalation paths in AI products.

How routing looks across our companies

The same gateway serves very different products, so the routing tables look quite different from company to company.

MerchNivo, the AI e-commerce employee for Shopify stores, runs many small tasks and a few larger ones. Tagging order notes, classifying customer messages and flagging unusual refunds are small-model work. The daily briefing, which has to weigh several signals and pick what deserves a merchant's attention first, sits on a mid-tier model. Imagine a store sees a spike in refunds on a single product overnight. Detecting the spike is arithmetic done in code, from the store's own data. Classifying the refund reasons is a small-model task. Writing the one-paragraph explanation that connects the spike to a recent supplier change mentioned in order notes is where a mid-tier model earns its cost. None of those steps asks a model to invent a number.

ZodiVela is a consumer product with many short interactions. Most of its tasks run on small models with strict length and content limits. Longer readings use a mid-tier model, and per-user rate limits keep usage predictable. The value of routing here is mainly cost: in a consumer product with credits and subscriptions, the difference between tiers shows up directly in the product's unit economics.

KeşifAtlası routes differently because accuracy matters more than speed. The eligibility decision itself never touches a model; it comes from the verified rules database. Explaining which rules apply to a person's case, in plain language and with references to rule identifiers, runs on a stronger model, because a weak explanation of a correct decision still misleads people. Simple tasks like normalising a free-text country name run small. See Rules engines for eligibility.

EduRelia adds another constraint: what can be sent at all. Routing rules there sit alongside data minimisation rules, so a feature's model choice is made after deciding which fields may leave the product. See Data minimisation for children in edtech.

Four products, four very different tables. The shared gateway handles the mechanics. Each company owns its own rules.

When a new model is released

Model providers release new models often, and prices change. A routing setup that cannot absorb that churn will either fall behind or lurch from one model to the next based on excitement.

Our process is deliberately boring:

1. One product with a relevant feature runs the new model against that feature's existing test set.
2. The four numbers are compared against the current primary.
3. If the new model is better on quality, cost or latency without losing on the others, the product owner changes the routing rule and updates the reason field.
4. The change is rolled out to a share of traffic first, with validation and fallback rates watched closely.
5. Other products evaluate independently. No product switches because another did.

Because routing is configuration, a switch is a change to one entry, not a code release. Because every rule has a test set, the evaluation takes hours rather than weeks. That is what makes model choice a routine operational decision instead of a project.

Common mistakes

A few patterns show up again and again when teams set up routing.

• Routing by product instead of by task. It hides the easy savings inside every product.
• No test set. Without one, every routing decision is a guess and every model switch is a gamble.
• Chasing the newest model everywhere. New is not the same as better for your task. Evaluate first.
• Ignoring retries in the cost. Cost per call flatters cheap models that fail often.
• Letting the model fetch or decide facts. It pushes you toward larger models to compensate for a design problem.
• Dynamic routing too early. It adds a moving part before you know whether the inputs actually vary in difficulty.
• No fallback. Every route needs a plan for when the model is slow, down or wrong. See Fallbacks and degraded modes for AI features.
• No owner. A rule nobody owns is a rule nobody revisits.

A short checklist for a new feature

When a team adds an AI feature to one of our companies, the routing part of the review is a few questions:

• What task type is this?
• What facts does the model receive, and where do they come from?
• What does a good output look like, and is there a test set of at least a few dozen real-shaped examples?
• What is the smallest model that passes it?
• What does it cost per useful outcome, including retries?
• What is the fallback, and what does the user see if everything fails?
• Who owns the rule, and what is the reason written next to it?

If the team can answer those, the routing rule is ready. If they cannot, the feature is not ready either, whatever model it uses. Caching can reduce the load further for repeated, low-variance requests, which we cover in Latency, caching and batching.

Summary

Model routing means matching each AI task to the smallest model that does it well, and writing that choice down as a rule with an owner, a fallback and a reason. We route by task type rather than by product, start every feature on a small model and move up only when a test set shows a real gap, and judge models by cost per useful outcome rather than cost per call. Fixed rules come first; dynamic routing and in-request escalation are added only where inputs genuinely vary and failures are detectable. Kept in configuration inside a shared gateway, routing lets each Oryvelon company absorb new models and price changes as routine decisions, while facts and decisions stay with each product's verified systems.

Questions and answers

What is model routing in AI products?

Model routing is the practice of sending each AI task to the model best suited to it, usually by sending simple, high-volume tasks to smaller models and reserving larger models for complex reasoning.

How do you decide whether a task needs a large model?

Run the task's evaluation set on the smallest candidate model first. If it meets the quality bar, it stays there; a larger model is only used when tests show a consistent gap that matters to users.

Should routing be dynamic or fixed per feature?

Most teams should start with fixed rules per feature because they are predictable and easy to audit. Dynamic routing based on input difficulty is worth adding only when a feature has clear, measurable variation in difficulty.

Published by Oryvelon, a digital company builder · About the group · Our companies · More in AI Operations

NextPrompt versioning and evaluation for production AI →