Routing and failover
Updated on 3 November 2026.
For every request the gateway decides three things, in this order:
- Which route applies: a Policy Engine route, an app routing group, or just the requested model.
- Which model and provider in that route: kept the same for the whole conversation so the provider's prompt cache keeps hitting.
- What to do when that provider fails: retry it, or move the request to a backup.
With the Policy Engine on for the LLM Gateway, the Routing Groups & Fallbacks control owns all three. Without it, the app's Routing tab does; see Routing without the Policy Engine.
Configure routing
Choose your gateway's mode. Classic app routing settings stay stored while the Policy Engine is on; published policy routes decide live traffic.
- Classic settings
- Policy Engine
Routing without the Policy Engine
Without the Policy Engine, routing is set per app:
Group names are unique across both group types and can shadow a real model name, so a group called gpt-4.1 reroutes existing code with no deploy. Custom routing tiers route only when Published; on Bedrock Runtime it covers Converse only.
The provider is picked as below. The choice is pinned per conversation exactly as described in Conversation pinning, and failover runs with its default settings.
- Is the model a routing group name?yes:A group memberFurthest below its weight, then pinned
- Does the request name a provider?provider, provider_label, X-Provider-Name or X-Provider-Labelyes:That providerNever failed over
- Does exactly one enabled provider serve the model?yes:That provider
- Do several enabled providers serve it?yes:One of them, at randomThen pinned
- None of the aboveError: the model is not on the app
from openai import OpenAI
client = OpenAI(base_url='https://guardrails-usa-2.quilr.ai/openai_compatible/', api_key='sk-quilr-xxx')
response = client.chat.completions.create(model='Group1', messages=[{'role': 'user', 'content': 'Hello!'}])
Which providers a group can mix
A group serves one API surface (its kind), but its members can come from any API family. With any-to-any formats, members that do not speak the group's format are reached by conversion.
- At least one member must be from the group's own family.
- While any own-family member is enabled and usable, only own-family members take traffic.
- Every member must be reachable from the group's surface; unsupported and coming soon pairs cannot be added.
- Realtime groups stay within OpenAI Realtime and Azure OpenAI Realtime.
When a provider fails
Without the Policy Engine, Failover runs with its defaults: the group's other members first, then the same model on the app's other providers. A request blocked by a guardrail is never failed over. Disabled providers are left out before the call.
Where it lives
Open Govern > Policy Engine > LLM Gateway > Controls > Routing Groups & Fallbacks. Use Start from a recipe for a common setup, or New route for another route.
Use Search routes, models or groups and the All, Limits, Split, Try in order, Content, and Watch only filters to find a route. Configure the scope and destination, check the readable rule summary, then Save to draft. Live traffic changes after Review & publish.
Routing limits apply across matching routes and during failover, regardless of route rank. Without a matching route, requests go to the model the client asks for.
From scripts and pipelines
Everything on this card can also be managed with the Management API, as typed resources: routes, weighted and ordered destinations, failover profiles, rules and default, routing constraints and model access. Admins never need to write QuilrQL for these: each API write is validated, turned into policy rules and published as one revision, with dry runs, ETags, revision history and restore.
Rules written in the console appear in API lists as read-only rows; API-managed routes and console rules are ordered as described in Rank and precedence.
While the Policy Engine is on, the app's Routing tab is frozen: app routing groups and custom routing are ignored. Converted app groups become policy groups scoped by Requested model; see What happens to classic settings. Provider credentials, endpoints and the enabled switch stay live and are looked up by provider label.
How a request is routed
Step 1: which route applies
- Does Allowed Models reject the requested model?yes:403 model_access_denied
- Does the request continue state held by one provider?previous_response_id, cachedContentyes:The credential that holds itNever moved
- Has this conversation already been given a destination?Keep destination routes, matched data routesyes:Same destinationUnless a higher route takes over
- Does a content route match?Highest in the list firstyes:Route's destination
- Does a Route to or Routing group match?Higher priority wins; a tie goes to Route toyes:Targets or weighted split
- None of the aboveRequested modelOne provider serves it, or one is picked and kept
- Rejected models on the Allowed Models card win over every route, including models reached only through routing.
- Priority decides between configurations. Content routes take their priority from their place in the list. Two overlapping routing rules cannot share a priority.
- Data routes win over the route that would otherwise apply when their priority is equal or higher. They are checked on every message.
- A destination that cannot serve the request's API (natively or by conversion) is skipped. If no destination of the route can serve it, the request uses normal routing.
Step 2: which model in the route
Members from the request's own API family are preferred. Members that can only be reached by conversion are used when no own-family member is available. Skipped members are left out and the remaining weights are rescaled.
Conversations and caching
Providers cache the start of a prompt (system prompt, tools, earlier turns). The cache belongs to one provider credential and model, and every move to another model starts a cold cache: full input price and slower first tokens. So routing decisions are made per conversation, not per request.
A gateway that routes each request on its own spreads one conversation across providers and pays for a cold cache every time it moves:
The LLM Gateway keeps the conversation where its cache is:
Cache-aware by design
How the gateway identifies a conversation
The first of these present on the request becomes the conversation key:
The key is also scoped to the Quilr key and the requested model, so a coding tool's small helper model is routed separately from its main model. Send a conversation ID header when you can: with option 5, two conversations from the same user that start identically share one key. The Identity & Network Trust card can require one.
Conversation pinning
When a routing group or a multi-provider model picks a member, that choice is pinned for the conversation after the response succeeds (a stream has to finish). Failed requests never create a pin. A pin lasts until the conversation has been idle for 1 hour.
A pinned conversation is routed again when:
- the configuration that chose it is edited (any change, including weights or priority), removed or expires,
- the routing group or API changes,
- the pinned provider is disabled, loses its credential, or can no longer serve the request's API,
- 3 requests in a row fail on it with a 5xx, 401, 403, 404, 408, 424 or 429 (requests blocked by QuilrAI guardrails do not count),
- a data route matches,
- or failover moves it, in which case the conversation is pinned to the backup and does not move back.
Route to needs no pin: its first available target is the same on every request.
Conversation behaviour
Each content route has a Conversation behaviour (called Switching in the complexity form). It decides whether a route is re-checked on later messages:
Data routes always re-check every message and, once matched, keep their destination for the rest of the conversation until a higher-priority data route matches or the rule changes.
A stored destination is dropped and routing runs again when its rule is edited, removed or expires, or when the destination can no longer be used.
Keep prompt cache warm
Provider caches expire after a few minutes without traffic. Conversations > Set up opens Keep prompt cache warm for chosen apps. Both options are off until you turn them on.
Never kept warm: requests with server-side or hosted tools (including web search), Responses conversation or background state, requests translated to another API, private inference, blocked requests and response-cache hits, and prompts below the minimum. A refresh never moves or extends a conversation pin.
routing.keep_cache_warm = { stop_after_idle_minutes: 30, min_prompt_tokens: 4000, extend_cache_lifetime: true, refresh_requests: true };
The Response cache on the Token Savings card is different: it answers a repeat of a non-streaming request from an earlier response, without calling the model. Keep prompt cache warm only keeps the provider's prompt cache alive.
Routing types
All of them apply on the Chat Completions, Anthropic Messages, Responses, Vertex AI and Bedrock Runtime surfaces. OpenAI Realtime supports routing groups only. Embeddings are not routed.
Start from a recipe opens a form for common setups: Send an app to a model, Primary model with a backup, Split traffic across models, Route by topic, and Route sensitive data to a private model. Complexity recipes include Small model for simple prompts, Three tiers: small / medium / large, and By message length only.
Route to
An ordered list of provider label and model pairs. The first target that is enabled, has a credential and can serve the request's API takes the request, whatever model the client asked for. The rest are its failover order.
- Choose Send an app to a model under Start from a recipe, then pick the apps or groups.
- Under Send to, keep Ordered targets. Pick a Provider label and a Model for target 1.
- Add target for each backup and drag to reorder.
- Save to draft, then Review & publish.
policy support_copilot_route priority 650 {
request where application.group_key == "support-copilot"
then { routing.targets = [
{ model: "gpt-4.1", provider_label: "openai_primary" },
{ model: "gpt-4.1", provider_label: "azureopenai_primary" }
]; }
}
Routing groups
A weighted split across models. Weights total 100.
- Choose Split traffic across models under Start from a recipe, then pick who it is for.
- Under Split, give it a Name, choose the Kind (Chat completion, Anthropic messages, Responses, Vertex AI, Bedrock or Realtime) and the Mode:
- New conversations: weights split new conversations.
- Split by tokens: weights split total tokens.
- Add model for each model and set its weight. Balance evenly splits them equally.
Because a conversation stays on the model it started with, request and spend shares can differ from the weights. The group is saved as policy-<name> and never affects an app routing group with the same name.
Content routes
Content routes pick the destination from what the conversation contains. They share one form, Add a content route: who it applies to, Match when, and Send to (Ordered targets or Routing group). A content route can only set a destination, its conversation behaviour, observe only and its own failover.
Request is about (description routing)
Describe the requests in a sentence or two (up to 300 characters). An AI classifier judges whether the latest user message matches.
- Long messages are classified on a bounded portion of the text; check long prompts with Test routing.
- Up to 32 descriptions are checked per request, highest priority first.
- With Keep destination, the classifier runs on the first message of a conversation; a result is reused while the latest user text is unchanged.
- If the classifier is unavailable or times out, the route is skipped and the request falls through to the next route. The skip is shown in Routing activity.
policy route_coding priority 720 {
request where application.method_type == "chat"
and about(content.last_user_text, "Programming help: source code, SQL queries, shell scripts, bugs or stack traces.")
then { routing.targets = [{ model: "claude-sonnet-4-6", provider_label: "anthropic_primary" }]; }
}
Complexity routing
Send simple prompts to a small model and hard ones to a large one. Start from a complexity quick add card or Set up complexity routing under Complexity levels.
- Application: pick the app, or leave empty for every app.
- When the app asks for (optional): limit to requested model names, for example
auto. - How to judge:
- AI reads the message (falls back to length): the classifier compares the latest user message with your level descriptions. If it is unavailable or unsure, the length bounds decide.
- Message length only: words or tokens in the latest user message decide. No AI call, no added latency or cost.
- Levels: for Low (and Medium with Use a Medium level) write what the level means and its upper bound in Words or Tokens. A message at or under the Low bound is Low, at or under the Medium bound is Medium, anything else is High.
- For each level choose Normal routing, One model or first available, or Split by %.
- Switching: see Conversation behaviour.
Without Complexity levels, Prompt complexity counts words over all user text in the request: 6 or fewer is Low, 7 to 10 is Medium, more is High.
policy cx priority 760 {
request where application.group_key == "support-copilot"
then { routing.complexity_levels = {
low: { description: "Greetings and short factual questions", max_tokens: 300 },
medium: { description: "Everyday drafting and summaries", max_tokens: 4000 },
high: { description: "Multi-step reasoning, design or long analysis" }
}; }
}
policy cx_low priority 759 {
request where application.group_key == "support-copilot" and content.context_complexity == "low"
then { routing.targets = [{ model: "claude-haiku-4-5-20251001", provider_label: "anthropic_primary" }];
routing.route_decision = escalate; }
}
// cx_medium and cx_high follow the same shape
For length only: { method: length, low: { max_words: 15 }, medium: { max_words: 120 }, high: {} }.
Data detected
Move a conversation to a model you trust when chosen data types appear, for example an on-premises model for secrets. Choose Route sensitive data to a private model under Start from a recipe, choose the data types, then the destination.
- Data routes are evaluated on every request and go first in the list by default.
- Once matched, the conversation keeps this destination.
- Best effort: if the scan cannot run, normal routing is used. Use Security guardrails to redact or block.
- Types used only for routing do not create findings, redact or block. Matches appear only in Routing activity.
Text contains
Route when the latest user message contains any of a list of words or phrases. Comma separated, not case-sensitive. * matches any text in between (refund*order), ? one character.
Client
Route by the coding tool named in the request headers, for example Claude Code, Codex, Cursor, Cline or GitHub Copilot. The client reports this itself; it is not an identity check.
Observe only
Every content route has an Observe only switch. The route is checked on every message and Routing activity records where matching conversations would go, while requests keep their normal destination. AI and data checks still run, so they still add latency. Observe only cannot be combined with a conversation behaviour or a failover setting.
Order
Under Order, place a route 1st, after another route, or Last. Higher in the list wins. New routes go last; data routes go first. Earlier and Later on the card move a route.
Budget used
Route by how much of a budget the requester has used, for example to a smaller model past 80% of their monthly spend. Write it in a policy with budget_used_pct("<budget id>"), compared with >=, >, <= or < against a number, or add conditions.budget to a route through the Routing and failover API. The value includes the requester's grants. In a policy, add routing.route_decision = per_message so it is checked on every message rather than kept for the conversation; API routes do this for you.
If the usage cannot be read, the rule does not match and normal routing applies. Route below 100%: a budget that is used up and enforced still refuses the request. Example: Move to a cheaper model as a budget runs low.
When a routing check fails
Content and data routes are best-effort routing, not enforcement. A check that cannot complete never blocks the request: it is routed as if that route did not match. Use Security guardrails when data must be blocked or redacted.
Failover
When the chosen provider fails, the gateway retries it or moves the request to a backup. Configure it under Failover defaults for everyone or a scope, or per route with Set failover for this route. A route's own failover setting wins; otherwise the highest-priority failover setting that matches applies; with none, the defaults below apply.
Settings
routing.failover = { enabled: true, scope: same_model, fallbacks: [{ model: "gpt-4.1", provider_label: "azureopenai_primary" }],
on: [timeout, connection, rate_limit, server_error], max_attempts: 3, same_target_retries: 2,
total_timeout_ms: 60000, attempt_timeout_ms: 0 };
A partial value is filled from the defaults, not merged with other failover settings.
Where a failed request goes
Backups are tried in tiers. A tier is used only when the tiers before it have no usable candidate left.
Every candidate must have an enabled credential, be allowed by Allowed Models, be inside a matched data route's destinations, and serve the request's API without dropping any of its features. A candidate that is the same provider account and model as one already tried is skipped. Before a backup is sent, its quotas and per-model limits are checked; a backup over its limit is skipped.
Other app models are ranked by App model ranking, then same API family, then same vendor, then closest price. Without Model capabilities, such a model is used only for requests with no tools, images or JSON schema, an estimated prompt under 8,000 tokens and requested output up to 4,096 tokens, and for streams only when the provider streams natively for the API.
Because Max tries is at most 3 and includes the first send, a request reaches at most 2 backups.
Retry or move
- Was it a QuilrAI block or a gateway error?Guardrail, quota, access, conversion, configurationyes:Returned to the clientNever retried
- Are Max tries or the Time budget used up?yes:Last provider's error
- Is the error in Fail over on, and is a backup left?A 429 with Retry-After of 2 s or less retries in place insteadyes:Next backupSame-model retry count resets
- Is it a timeout, connection error, 429 or 5xx in Fail over on, with same-model retries left?yes:Retry the same model0.5 s, then 1 s, or Retry-After up to 10 s
- None of the aboveThe provider's error, unchanged
- Fail over on controls both moving and same-model retries: an error left out of it is returned to the client on the first try.
- A retry is made only if its delay fits in the remaining time budget.
- When every try fails, the client gets the last provider's error with its own status, body and
Retry-After, exactly as if failover did not exist. - When nothing could be sent (every candidate marked down or over quota, or the time budget already used up), the client gets
503 all_targets_unavailablewithx-quilr-error-source: gatewayand, when known,Retry-After. - With Provider content filter on, a successful response that the provider refused or filtered is also sent to another provider.
Requests that never move
These are only retried on the same provider:
- requests that continue provider-held state:
previous_response_id, Responsesconversationorbackground, GeminicachedContent, stored prompts, containers and MCP servers held by the provider, - requests with hosted or server-side tools, or that reference provider file ids,
- requests that name a provider with
provider,provider_label,X-Provider-NameorX-Provider-Label, - requests served by a private inference edge.
For stateful requests, a retry is made only when the provider certainly did not run the first try (the connection was never opened, or a 429). Embeddings fail over only to the same model on another connection. OpenAI Realtime and Assistants are not retried or failed over by the gateway.
Streaming
For a stream, the gateway waits for the provider's first event before it starts responding to the client. An error, timeout or error event before that can still retry or fail over. Once the first event reaches the client, the stream is committed to that provider and a later error is passed through.
Provider health and cooldown
The gateway tracks the health of every provider credential and model it calls:
A cooldown only affects requests whose Fail over on includes the error that caused it, and each kind of error cools down on its own: a 401 cooldown does not hide a later run of 5xx errors from requests that only fail over on 5xx.
During an outage you can also mark a provider down for every app, with an expiry, and reset cooldowns once it recovers, with the Incidents API.
A cooldown routes around a provider; it never refuses a request. A cooling target is skipped while a healthy backup exists. When its cooldown ends, one request at a time is sent to it as a probe; success clears it, failure starts a longer cooldown. If every candidate is cooling, including a route with a single target, the request is still sent, to the one that recovers first.
Failover, pinning and quotas
- After a request moves from A to B and succeeds, the conversation is pinned to B and stays there until it goes idle. There is no automatic failback.
- A content-route decision stored for the conversation is not changed by a failover.
- Quotas count one logical request. Each provider that was actually sent to is charged for its own usage at its own price; the app-level request count is charged once.
See what happened
The request log lists every try with its provider, model, tier, outcome, status, latency and spend. The failover preview shows where an app's failed requests would go.
Test and verify
Test before you publish
Use Test routing in the Routing Groups & Fallbacks panel to check a routing scenario. For a broader policy check, open Test > Try a situation in the LLM Gateway Policy Engine, describe the request, and review the rules that decide its settings. Test both a request that should route and one that should keep its requested model.
Use Replay activity to compare the candidate rules with recent recorded activity before publishing. See Author, simulate and publish.
Check live traffic
Every successful gateway response carries:
curl -sD - -o /dev/null https://guardrails-usa-2.quilr.ai/openai_compatible/v1/chat/completions \
-H "Authorization: Bearer sk-quilr-xxx" -H "Content-Type: application/json" \
-d '{"model": "gpt-4.1-nano", "messages": [{"role": "user", "content": "hi"}]}' | grep -i x-quilr
The panel's Activity tab shows per route and app over 24 h, 7 d or 30 d: conversations, requests, spend (including keep-warm refreshes), cache read, added time, failures, upgrades and fallbacks. Hover a route to see its route id.
Rules and limits
- The requested model must be allowed for the app. A name such as
autothat exists only for routing is rejected withmodel_access_deniedunless the app's allowed models include it. - Under the Policy Engine, a client that sends an app routing group name the policy does not define gets
403 policy_routing_group_not_defined, orpolicy_routing_group_wrong_surfacewhen the group is for another API. - Two overlapping routing rules cannot share a priority.
- Upgrade when needed is only for complexity routes. Observe only cannot be combined with a conversation behaviour or failover.
- Request is about cannot be negated.
- On deployments without GPU models, the AI classifier is never called; complexity routes use the length bounds.
- A spend budget needs a price for every model it can reach, including failover backups; see Rate, token and budget limits.
Related
- Routing and failover API - manage routes, destinations and failover from scripts, without QuilrQL
- Incidents API - outage mark-downs, cooldown resets and pin drops
- Any-to-any formats - how a chosen provider is reached
- Gateway access and allowed models
- Token saving - prompt compression
- Response cache - answer repeat requests without calling the model
- Conversation compaction - keep long conversations within the model's context
- Author, simulate and publish
- Policy Engine overview