Route requests with a Dynamic LLM

A Dynamic LLM is a Model Hub entry that has no provider behind it. It carries a routing configuration instead, and at call time it resolves to one of your real models — either by evaluating ordered rules against signals taken from the request, or by asking a decider model to classify the request into a label.

Callers address it like any other model: same base URL, same model field, one key. Nothing changes on the client side when you later re-point the router at different models.

Who this is for

The click-paths are written so an operations lead can build a router without touching code. Each section then gives the engineer the payload, the exact signal semantics and the trap.

What you get

• One model key — say smart-router — that callers put in model instead of choosing a model per request.
• A recorded decision for every request: which rule or label fired, which model actually ran, and the signals that produced the choice. Readable on the model's Routing tab.
• An optional fallback model that runs when the chosen model throws.

Strategy	How the target is chosen	Extra cost per request
Rule-based	Ordered rules evaluated against signals computed from the request body. First match wins.	None — the rules are arithmetic on the request.
Model-based (decider)	A classifier model reads the latest user message and returns one of your labels; the label picks the target.	One extra model call, billed to the decider model.

Rule-based routing is deterministic and free. Reach for the decider only when the split you need depends on meaning rather than on shape — "is this a support question or a code question" — and accept that you are paying a model call and adding its latency to every request.

Before you start

You need	Where
At least two LLM models in the active project	Create your first model
Pricing filled in on the default model, if you want to route on cost	Model Hub → the model → Edit model
An API token to call the router with	Settings → API Tokens → Create Token

Routing targets are drawn from LLM models in the active project only. Models of any other category — embedding, rerank, speech-to-text, text-to-speech, OCR — are not offered, and neither are other Dynamic LLMs. If the project has no LLM models, the form says so and there is nothing to select.

Create the router

Open Model Hub in the left nav, then Dynamic LLM in the sub-nav. That item is a shortcut: it opens the model list at ?create=dynamic with the Create Dynamic LLM form on top of it — "Route each request to a different model by rules or by a decider model." Closing the form drops the create parameter and leaves you on the list.

The form has four sections and a footer that reads 3 of 3 ready once the checklist — Name set, Default model chosen, Rules configured (or Decider & labels configured) — is complete.

1. Identity

Field	Notes
Display name	Required.
Key	Leave blank to auto-generate from the display name. This is the value callers put in model. The form locks the field once the router exists — its hint reads "Key is immutable after creation."
Description	Optional.

2. Strategy

Pick Rule-based or Model-based (decider). The choice decides whether section 4 is Rules or Decider; everything else is identical.

3. Default & fallback

Field	When it runs
Default model	Required. Runs when no rule matches, and whenever the decider fails or returns something that does not map to a label.
Fallback model	Optional. Runs only when the chosen model throws.

Both selects list your LLM models as Display name · key.

4a. Rules (rule-based)

Each rule card has a Label (free text, used in the decision badge and the reason line), a Route to target model, a Match control set to All conditions or Any condition, and one or more conditions. Add condition and Add rule extend the form.

Rules are evaluated top to bottom and the first match wins. Order them from most specific to least.

These are the eight signals, exactly as the form lists them:

Signal	Type	How it is computed
Estimated input tokens	number	Total characters of the text content across all messages, divided by 4 and rounded up. No tokenizer is involved. Image parts and tool definitions contribute nothing.
Message count	number	Number of entries in messages, all roles included.
Last user message length	number	Character count of the most recent user message.
Estimated cost in USD (at default model pricing)	number	Estimated input tokens priced at the default model's input rate, plus max_tokens × its output rate when the caller sent max_tokens. Output size is unknowable before the call, so without a cap the estimate is input-only.
Request uses tools	boolean	tools is a non-empty array, or tool_choice is present and not "none".
Structured output requested	boolean	response_format is present and not null.
Request has images	boolean	Any message part typed image_url, image or input_image, or carrying an image_url key.
Keyword in last user message	text	Substring or regular-expression match against the most recent user message. Both are case-insensitive.

And the operators. The form offers only the ones valid for the selected signal; changing the signal resets the operator and clears the value.

Operator	Applies to
> greater than, ≥ at least, < less than, ≤ at most, = equals, ≠ not equals	the four numeric signals
is true, is false	the three boolean signals
contains, matches (regex)	Keyword in last user message

A typical two-rule router: send anything with images or tools to the capable model, send short conversations to the cheap one, and let everything else fall through to the default.

The cost signal is conditional

Estimated cost in USD is computed only when a rule actually references it, and only under the rule-based strategy. If the lookup of the default model fails, or the default model has no pricing, the signal stays unset — and an unset signal never satisfies a condition, whichever operator you used. A default model left on the create form's zero pricing makes every estimate exactly 0, so > 0.01 never fires and < 0.05 always does.

Route on who is calling

Available from the next Console release

Caller conditions, pools, guards, shadow rollout and routing through agents arrive with the next Console release.

Besides the shape of the request, a rule can look at who is asking and through which door. These are the caller signals. They are listed under the Caller group of the signal picker and take a list of values:

Signal	Matches against	Operators
Channel	Where the call came from: dashboard, portal, api, embed, a2a, schedule or playground	in, not_in
User	The signed-in user's id	in, not_in
Group	Any of the caller's groups (resolved from the user when a rule needs it)	in, not_in
Agent	The key of the agent that is running	in, not_in
Portal	The key of the portal the call came through	in, not_in
API token	The id of the API token used	in, not_in
Project	The project id	in, not_in

In the API the signals are channel, userId, groupIds, agentKey, portalKey, apiTokenId and projectId. The value is a list: an array, or a comma-separated string. A condition accepts at most 100 values, and an unknown channel name is rejected when you save.

Missing context never matches

If the caller did not supply a field, a condition on it is false, whichever operator you used, including not_in. A rule such as "userId not in the finance team" therefore does not fire for a call that carries no user. Put the catch-all in the default model instead of relying on a negative condition.

Rules still run top to bottom and the first match wins, so put the narrow caller rules first. A router holds at most ten rules and a rule at most five conditions; "all" or "any" matching works exactly as for request signals.

Example: send the support agent's traffic from the portal to the cheap model, and keep everyone else on the default.

{
  "label": "support portal",
  "targetModelKey": "gpt-small",
  "matchType": "all",
  "conditions": [
    { "signal": "channel", "operator": "in", "value": ["portal"] },
    { "signal": "agentKey", "operator": "in", "value": ["support"] }
  ]
}

Other signals need a lookup and are only computed when a rule references them: the complexity score (from a decision model you attach in the router form), the conversation's spend so far, the share of the cost-guard budget used, the predicted output tokens and the predicted output-to-input ratio. If the value cannot be computed, a condition on it does not match either.

4b. Decider (model-based)

Field	Notes
Decider model	Required. In releases that have decision models, a new decider must be a decision-category model (it answers one choice question with a probability per label; an optional minConfidence from 0 to 1 sends low-confidence answers to a belowConfidence target). A decider that is a plain LLM model is accepted only while it is the decider the router already had. Before that, any LLM model in the project. It is called once per request, at temperature 1 with a 256-token cap.
Prompt override	Optional. Replaces the built-in classification system prompt in full.
Label	Required per card. The exact string the decider is expected to return.
Route to	Required per card. The model that label routes to.
Description	Free text that "Helps the decider tell labels apart" — it is injected into the built-in prompt next to the label.

Add label adds cards; two are present by default.

The built-in prompt sends the decider a system message listing every "label": description pair and instructing it to answer with the label alone, plus one user message containing the latest user message text (or (empty request) when there is none). The answer is matched back to a label case-insensitively: an exact match first, then a substring match, so Category: simple still resolves to simple.

A prompt override replaces the label list too

The override is used instead of the whole built-in system prompt, not merged with it. If you supply one, you must list your labels in it yourself. Leave it blank unless you have a concrete reason — an override that omits the labels leaves the decider with no vocabulary, so every answer fails to match and every request lands on the default model.

The decider sees the latest user message and nothing else. Not the system prompt, not earlier turns, not tools, not images. Route on those with rules instead.

Press Create Dynamic LLM. The router appears in the model list with a teal dynamic badge in the Type column instead of a category badge.

Watch the decisions

Open the router from the model list. A Dynamic LLM's detail page carries a Routing tab that regular models do not have, and no Playground tab — there is no provider runtime to play against.

The tab's left pane lists recent routing decisions:

Column	Contents
When	Relative timestamp.
Routed to	The model key that actually ran.
Decision	rule, model, default or fallback, with the matched rule label or decider label appended. fallback renders as a warning badge.
Reason	The full sentence explaining the choice — this is the column that matters.
Latency	Round-trip for the router row.

Rows are clickable and open the full log entry, which carries the signals that were computed. The right pane restates the live configuration: strategy, default, fallback, and every rule or label with its target. Before any traffic arrives the tab reads "No routing decisions recorded yet. Send a request to this model key to see how it routes."

Router decisions are also visible on the Logs tab, recorded under the route chat.completions.router.

Read the reason column, not the error rate

When the decider call fails, or returns a label you do not recognise, the router does not error. It routes to the default model and writes the explanation into reason — Decider failed (…); used default model or Decider returned an unrecognized label "…"; used default model. The request succeeds, the caller sees nothing unusual, and your error rate stays flat while every request quietly runs on the default model. A router whose decisions are overwhelmingly default is a broken router, not a quiet one.

The same thing over the API

Calling the router

Nothing special — it is a model key on the standard OpenAI-compatible endpoint:

curl -s https://<host>/api/client/v1/chat/completions \
  -H "Authorization: Bearer cpeer_…" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "smart-router",
    "messages": [{ "role": "user", "content": "Summarise this ticket in one line." }]
  }'

The response body is the child model's response, passed through untouched. Its model field is therefore the child key, not the router key — that is how a caller learns which model answered. The same holds for streaming chunks.

GET /api/client/v1/models lists routers alongside real models, with owned_by set to dynamic.

Passing your own request_id in the request body makes the router's decision row and the child's usage row share one id, which is the easiest way to correlate them. Without it the router row records the child's id in childRequestId.

Creating a router

POST /api/client/v1/models/dynamic     # Bearer cpeer_… token
POST /api/models/dynamic               # browser session (what the form uses)

Two endpoints, one body

Both take the body below and return 201 {"model": …}. The client endpoint authenticates with Authorization: Bearer cpeer_… and creates the router in the token's project. /api/models/* authenticates with a browser session and the active_project_id cookie — a cpeer_ token cannot drive it. Only the dashboard endpoint checks the project's model quota, returning 429 when it is full.

Body — rule-based:

{
  "name": "Smart router",
  "key": "smart-router",
  "description": "Cheap by default, capable when the request needs it",
  "dynamic": {
    "strategy": "rule-based",
    "defaultModelKey": "gpt-4o-mini",
    "fallbackModelKey": "claude-haiku",
    "rules": [
      {
        "label": "complex",
        "targetModelKey": "gpt-4o",
        "matchType": "all",
        "conditions": [
          { "signal": "inputTokensEst", "operator": "gt", "value": 4000 },
          { "signal": "hasTools", "operator": "isTrue" }
        ]
      },
      {
        "label": "vision",
        "targetModelKey": "gpt-4o",
        "matchType": "any",
        "conditions": [{ "signal": "hasImages", "operator": "isTrue" }]
      }
    ]
  }
}

Body — model-based:

{
  "name": "Intent router",
  "dynamic": {
    "strategy": "model-based",
    "defaultModelKey": "gpt-4o-mini",
    "decider": {
      "modelKey": "gpt-4o-mini",
      "labels": [
        { "label": "simple", "description": "Small talk, short factual questions", "targetModelKey": "gpt-4o-mini" },
        { "label": "complex", "description": "Multi-step reasoning, code, analysis", "targetModelKey": "gpt-4o" }
      ]
    }
  }
}

Config fields:

Field	Required	Notes
strategy	yes	rule-based or model-based.
defaultModelKey	yes	Model key, not an id.
fallbackModelKey	no	Model key.
rules[]	rule-based	At least one. Each needs targetModelKey and at least one condition. matchType defaults to all.
decider.modelKey	model-based	Required.
decider.labels[]	model-based	At least one; each needs label and targetModelKey.
decider.promptOverride	no	Replaces the whole classification prompt.

Rejections come back as 400 with the failing rule stated — rule-based routing requires at least one rule, every rule needs a targetModelKey, the decider needs at least one label, and so on.

Boolean conditions carry no value; a numeric condition's value is coerced with Number(), so anything that does not parse as a number never matches; a keyword condition's value is the substring or pattern.

Editing a router

PUT /api/client/v1/models/:id     { "settings": { "dynamic": { … } } }
PUT /api/models/:id               { "settings": { "dynamic": { … } } }

settings is merged shallowly, so dynamic is replaced wholesale — send the complete object, not a patch. Neither path re-runs the create-time validator, so a malformed config written here is accepted and only surfaces at call time. The Edit model action on a router opens the same Edit Dynamic LLM form rather than the standard model edit page, which is the safe way to do it.

Pools, guards, shadow rollout and analytics

A rule, a decider label or the default can point at one fixed model or at a pool: up to ten candidate models, each with a quality tier, and a policy that picks one per request.

Pool policy	Picks
Best under cost cap (best-under-cap)	The highest-tier candidate whose estimated cost fits the per-request cap. Without a cap, the highest tier.
Cheapest (cheapest)	The lowest estimated cost. Use it for equivalent models, such as the same model on several providers or regions.
Token profile (token-profile)	The lowest expected cost from the predicted output-to-input ratio, each model's verbosity and prompt-cache stickiness.

A candidate that lacks a capability the request needs (tool calls, images) or whose context window is too small is skipped, and the skip reason is recorded with the decision.

Cost guards apply after the target is chosen and can only move a request to a cheaper model:

Guard	Effect
maxCostPerRequestUsd	Per-request estimated cost cap.
conversationBudgetUsd	Downgrade once the conversation's realized spend through this router reaches the amount.
budget	A rolling window of 1 to 168 hours with a USD limit. Downgrades from downgradeAtPct (default 80), and at 100% either keeps serving on the cheapest model or rejects the request (onExceeded).
economyModelKey	The cheap model a fixed-model target downgrades to.
sticky, switchMarginPct	Token profile only: keep a conversation on its previous model for the prompt cache (default on), and switch away only for at least this percentage of saving (default 15).

Shadow rollout. Set mode to shadow and the router evaluates pools and guards and logs what it would have chosen, while traffic keeps going to each target's primary model (the fixed model, or the first candidate of a pool). In enforce mode (the default), canaryPercent (0 to 100, default 100) is the share of conversations that get the enforced decision; the rest run as shadow. Use shadow first, read the analytics, then enforce.

Analytics. A router's routing analytics are served by the dashboard API at GET /api/models/{id}/routing-analytics?days=7 (1 to 90 days, default 7). Each routed call also writes a decision row that records the strategy, the rule or label that fired, the pool estimates, the guard that changed the choice, the shadow decision, and the cost the same tokens would have had on the baseline model (baselineModelKey, default the default model). That is how savings are measured.

Agents run through the router

An agent can use a Dynamic LLM as its main model or as one of its additional models. Each model call of the run goes through the same resolution as a gateway request, so rules (including caller conditions), the decider, pools, guards and shadow mode all apply, and each turn and step is routed on its own. The caller context of an agent run carries the channel (from the run's source), the agent key, the project, the user (when a real person started it) and the API token id.

Chaining and the depth cap

Routing recurses, with a hard cap of 3. A router resolving to another router increments the depth, and the fourth level throws Dynamic routing depth exceeded (3) resolving model "…". The decider call consumes one level of that depth too.

The form never lets you build a chain — routers are excluded from the target lists — so this only happens through the API. A router that resolves to itself is caught separately: the target is rewritten to the default model, and if the default is the router, the request fails unless a fallback model is configured.

What will bite you

A router prices at zero

A Dynamic LLM is created with inputTokenPer1M, outputTokenPer1M and cachedTokenPer1M all set to 0, and there is no pricing form for it. The cost of a routed request is attributed to the child model, which logs its own usage row at its own prices. Two consequences:

• The router's row mirrors the child's token counts for traffic accounting. Tokens therefore appear twice across the usage rollups — once against the router at zero cost, once against the child at real cost. Cost never doubles; token totals summed across all models do. Filter by model key.
• Budget consumption is priced from the model key the caller addressed. Because that is the router, and the router prices at zero, requests sent to a router do not draw down a budget. Budgets are still checked before the call; they simply are not incremented by this traffic.

The fallback model only catches throws

fallbackModelKey fires when the chosen child model raises an error — provider outage, bad credentials, unknown model key. A slow answer does not trigger it. A truncated, low-quality or off-topic answer does not trigger it. There is no latency budget and no quality check. If the fallback also throws, the error is logged against the router and propagated to the caller.

Guardrails bound to the router never run	Routing resolves before any guardrail work, so only the child model's input and output guardrails apply. Bind them there.
Semantic cache on the router never runs	Same reason. Enable it on the child models.
Agents and routers	Released versions cannot use a router as an agent model. From the next Console release agent runs route through the router, see Agents run through the router.
No Playground tab	There is nothing to invoke locally; exercise the router over the control plane.
Streaming rows carry no tokens	For a streaming request the router's decision row is logged with empty usage. The child still logs its own.
An invalid regex is silently false	A matches (regex) condition with a broken pattern evaluates to false rather than raising.
Rules are ordered	First match wins, and a broad rule placed first makes everything below it dead.
Evaluation targets work	An evaluation target pointing at a router runs through the control plane path and routes normally, so you can A/B a router against a fixed model on the same dataset.

Where to go next

• How to optimize token usage — measure first, then decide what to split off onto a cheaper model
• Model Hub — models, keys and pricing
• Model Inference — the control plane, honoured parameters and error shapes
• Evaluation & Analysis — proving the cheap model is good enough before you route traffic to it