Use case

Dynamic LLM Routing By Request And Caller

Give applications one stable model key and let Console pick the concrete model per request, by prompt content, size, channel, or who is asking.

• Stack Console + OpenAI SDK
• Pattern Platform · Routing
• Read ~6 min

Overview

Applications usually want one model name in their code, while the platform team wants cheap models for easy traffic and strong models for hard traffic. A Dynamic LLM is a virtual chat model in Console: clients call it like any other model key, and Console decides per request which concrete model answers.

When to reach for this use case

Use it when one model key is shared by several apps, channels, or user groups and you want routing and cost decisions to live in Console rather than in application code.

Run the example

Runnable code for this pattern is public on GitHub:

• console-examples / 02-dynamic-llm-routing: A rule-based Dynamic LLM that sends different prompts to different models, and prints which model answered.

Architecture

Console owns the router. A Dynamic LLM has a strategy and a list of target models in the same project.

• Rule-based: ordered rules, first match wins. Each rule has conditions combined with matchType (all or any) and points at a model (targetModelKey) or a pool. If nothing matches, defaultModelKey is used, and fallbackModelKey is used if the chosen model errors.
• Model-based: a decider model classifies the request into one of your labels, and each label maps to a model or pool.

Rules can test request signals such as inputTokensEst, hasImages, hasTools, messageCount and keyword, and caller conditions such as channel, userId, groupIds, agentKey, portalKey, apiTokenId and projectId. Caller conditions accept only in and not_in. Optional guards cap cost per request, and mode: "shadow" logs decisions without enforcing them.

1. Create The Router

Create the Dynamic LLM in the Console UI (Models, then Dynamic LLM) with the rule-based strategy and a config like this:

{
  "strategy": "rule-based",
  "defaultModelKey": "balanced-model",
  "fallbackModelKey": "balanced-model",
  "rules": [
    { "label": "images-to-vision", "matchType": "all", "targetModelKey": "vision-model",
      "conditions": [{ "signal": "hasImages", "operator": "isTrue" }] },
    { "label": "refund-keyword-to-premium", "matchType": "all", "targetModelKey": "premium-model",
      "conditions": [{ "signal": "keyword", "operator": "contains", "value": "refund" }] },
    { "label": "api-long-prompts-to-strong", "matchType": "all", "targetModelKey": "strong-model",
      "conditions": [
        { "signal": "channel", "operator": "in", "value": ["api"] },
        { "signal": "inputTokensEst", "operator": "gt", "value": 300 }
      ] },
    { "label": "api-short-to-fast", "matchType": "all", "targetModelKey": "fast-model",
      "conditions": [{ "signal": "channel", "operator": "in", "value": ["api"] }] }
  ]
}

The model keys are placeholders for models in your project. Console validates the config on save: known signals, existing LLM models, and no router targeting itself.

2. Call It Like Any Model

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: `${process.env.CONSOLE_BASE_URL}/api/client/v1`,
  apiKey: process.env.CONSOLE_API_TOKEN,
});

const res = await client.chat.completions.create({
  model: "smart-router",
  messages: [{ role: "user", content: "Summarise this ticket in one line." }],
});

console.log(res.model); // the concrete model that answered

response.model is the provider model id of the concrete model that answered, not the router key.

3. Know What The Router Sees

Requests that arrive over the API token path carry the caller context channel = "api", apiTokenId and projectId. They carry no userId or groupIds, so a group rule never matches there. Group and user rules fire for callers that have an identity, such as the dashboard playground, portals, and agent runs. A rule on a field the caller did not supply simply does not match.

The routing reason is not returned to the client. It is stored on the router's usage log row, so use the Models logs and usage views in Console to see the matched rule, and the per-model usage analytics to confirm the traffic split.

Result

One model key in application code, with routing policy, fallbacks, and cost guards managed centrally.

Related use cases

Platform · Gateway

Quota-Aware LLM Gateway

Run application traffic through Console with project-scoped quotas, model routing, and request-level visibility for cost control.

Apps · Platform

AI-Powered Applications

Integrate AI capabilities into existing applications with OpenAI-compatible APIs, provider routing, and type-safe SDKs.

Platform · Governance

Enterprise AI Governance

Enforce guardrails, track usage, manage projects, and observe all AI operations from a single control plane.

See every pattern on the use case index, or browse all Cognipeer libraries.