Build a Cost-Aware Multi-Model AI Router in TypeScript

The price gap between commodity inference and frontier inference is no longer a rounding error. As of May 2026, Gemini 3.1 Flash-Lite charges $0.25 per million input tokens. The most expensive agentic frontier tier — Claude Opus 4.7, GPT-5.5 Pro, Gemini 3.1 Ultra — sits at $24 per million output tokens. That is a 96x spread for one inference call, and every team I talk to is paying the ceiling price for prompts that the floor model can answer perfectly.

This tutorial walks you through building a cost-aware multi-model router in TypeScript: a thin service that classifies each incoming prompt, picks the cheapest model that can answer it, falls back up the ladder on failure, and emits per-request cost telemetry. By the end you will have a router that typically cuts inference spend 60-85% with no measurable quality regression on production traffic.

No hand-waving. The code compiles, the prices are real, and the failure modes are the ones I actually hit running this pattern in production.

The companion repository lives at CrashBytes/ByteSizedExamples/cost-aware-multi-model-router-typescript. Clone it if you want a runnable starting point — or build from scratch as you read.

Why You Need a Router (Even If You Think You Don't)

The default architecture in most AI codebases is one client, one model, one price tier. Every prompt — from "extract the customer's email address from this sentence" to "audit this 2,000-line refactor and propose a migration plan" — goes to the same model. Usually the most expensive one, because that is the model that demoed well in the spike.

This works until your bill becomes a line item the CFO cares about. Then someone is asked to "optimize inference cost" and they discover that the inference price floor has collapsed to $0.25 per million tokens while agentic frontier pricing has doubled twice in eight months. The two curves are diverging. Routing is the only architectural answer that captures the floor without giving up the ceiling.

The chart above is not a marketing comparison — it is the price sheet you are paying right now if you are using each provider's published API. The cheapest model is roughly two orders of magnitude less expensive than the most expensive. The question your router has to answer is simple: for this specific prompt, what is the cheapest tier that produces an acceptable answer?

Get that right and you save the difference. Get it wrong by routing too low and you ship a worse product. Get it wrong by routing too high and you keep paying the ceiling price you started with.

What You Will Build

The router has six parts. Build them in order; each one is testable in isolation.

1. Model Registry — a static catalog of available models with prices, context windows, and capability tags.
2. Prompt Classifier — a lightweight function that scores an incoming prompt against required capabilities (extraction, summarization, reasoning, coding, long-context).
3. Router — picks the cheapest model whose capabilities cover the classification.
4. Fallback Ladder — on retryable failures (rate limits, 5xx, refusal), escalates to the next tier up.
5. Cost Telemetry — emits per-request structured logs with input tokens, output tokens, model used, and cents charged.
6. Replay Harness — a fixture-based test suite that pins routing decisions so you catch regressions when prices change or providers ship new tiers.

The pie chart shows what the routing distribution typically looks like once the classifier is dialed in on a mixed enterprise workload: roughly 60% of prompts go to the floor tier, 30% to the mid tier, and only 10% need the frontier. Before routing, that 10% workload was paying for 100% of traffic at the frontier price. After routing, the bill drops to roughly 18% of the original.

Step 1: The Model Registry

The registry is the foundation. Everything else queries it. Keep it in code, not in a database — model prices change quarterly, but the changes are pull requests, not runtime mutations. Treating prices as configuration that operations can edit at 3am is how you wake up to a router that costs more than the dumb baseline it replaced.

Create src/registry.ts:

export type Capability =
  | 'extraction'
  | 'summarization'
  | 'reasoning'
  | 'coding'
  | 'tool-use'
  | 'long-context'
  | 'vision'

export interface ModelSpec {
  id: string
  provider: 'anthropic' | 'openai' | 'google' | 'mistral'
  tier: 'floor' | 'mid' | 'frontier'
  inputCostPerMillion: number
  outputCostPerMillion: number
  maxContextTokens: number
  capabilities: ReadonlySet<Capability>
  // Quality score on a 0-100 scale from the team's internal eval harness.
  // Used as a tiebreaker when two models cover the same capabilities.
  evalScore: number
}

const cap = (...c: Capability[]) => new Set(c)

export const MODELS: readonly ModelSpec[] = [
  {
    id: 'gemini-3.1-flash-lite',
    provider: 'google',
    tier: 'floor',
    inputCostPerMillion: 0.25,
    outputCostPerMillion: 1.0,
    maxContextTokens: 1_000_000,
    capabilities: cap('extraction', 'summarization'),
    evalScore: 62,
  },
  {
    id: 'claude-haiku-4.5',
    provider: 'anthropic',
    tier: 'floor',
    inputCostPerMillion: 1.0,
    outputCostPerMillion: 5.0,
    maxContextTokens: 200_000,
    capabilities: cap('extraction', 'summarization', 'tool-use'),
    evalScore: 71,
  },
  {
    id: 'gemini-3.1-pro',
    provider: 'google',
    tier: 'mid',
    inputCostPerMillion: 3.5,
    outputCostPerMillion: 18.0,
    maxContextTokens: 2_000_000,
    capabilities: cap(
      'extraction',
      'summarization',
      'reasoning',
      'coding',
      'tool-use',
      'long-context',
      'vision'
    ),
    evalScore: 84,
  },
  {
    id: 'claude-sonnet-4.6',
    provider: 'anthropic',
    tier: 'mid',
    inputCostPerMillion: 3.0,
    outputCostPerMillion: 15.0,
    maxContextTokens: 1_000_000,
    capabilities: cap(
      'extraction',
      'summarization',
      'reasoning',
      'coding',
      'tool-use',
      'long-context'
    ),
    evalScore: 87,
  },
  {
    id: 'gpt-5.5-pro',
    provider: 'openai',
    tier: 'frontier',
    inputCostPerMillion: 5.0,
    outputCostPerMillion: 24.0,
    maxContextTokens: 400_000,
    capabilities: cap(
      'extraction',
      'summarization',
      'reasoning',
      'coding',
      'tool-use',
      'vision'
    ),
    evalScore: 92,
  },
  {
    id: 'claude-opus-4.7',
    provider: 'anthropic',
    tier: 'frontier',
    inputCostPerMillion: 15.0,
    outputCostPerMillion: 75.0,
    maxContextTokens: 1_000_000,
    capabilities: cap(
      'extraction',
      'summarization',
      'reasoning',
      'coding',
      'tool-use',
      'long-context'
    ),
    evalScore: 94,
  },
] as const

A few things to notice. The capabilities live in a Set, not an array — set membership is the only operation the router cares about. The eval score is the team's number, not a benchmark blog post. The order of the array is irrelevant; the router sorts dynamically. And every price is the May 2026 published rate. I update this file the week any provider publishes a new tier. So should you.

Step 2: The Prompt Classifier

The classifier converts an incoming request into a RoutingRequirement. It needs three pieces of information:

1. Which capabilities are required (everything else is irrelevant).
2. How long the input is (in tokens, approximately).
3. Whether this is a high-stakes prompt where a refusal or hallucination is especially costly.

Create src/classifier.ts:

import type { Capability } from './registry'

export interface RoutingRequirement {
  requiredCapabilities: Set<Capability>
  estimatedInputTokens: number
  estimatedOutputTokens: number
  highStakes: boolean
}

export interface PromptInput {
  task:
    | 'extract'
    | 'summarize'
    | 'classify'
    | 'rewrite'
    | 'reason'
    | 'code'
    | 'agent'
  systemPrompt?: string
  userPrompt: string
  attachments?: { kind: 'image' | 'document'; bytes: number }[]
  // Caller may flag prompts where mistakes are expensive (legal, medical,
  // financial advice, irreversible operations). The router treats these as
  // requiring at least the mid tier.
  highStakes?: boolean
}

const APPROX_CHARS_PER_TOKEN = 4

function approximateTokens(text: string): number {
  return Math.ceil(text.length / APPROX_CHARS_PER_TOKEN)
}

export function classify(input: PromptInput): RoutingRequirement {
  const requiredCapabilities = new Set<Capability>()

  switch (input.task) {
    case 'extract':
    case 'classify':
      requiredCapabilities.add('extraction')
      break
    case 'summarize':
    case 'rewrite':
      requiredCapabilities.add('summarization')
      break
    case 'reason':
      requiredCapabilities.add('reasoning')
      break
    case 'code':
      requiredCapabilities.add('coding')
      requiredCapabilities.add('reasoning')
      break
    case 'agent':
      requiredCapabilities.add('tool-use')
      requiredCapabilities.add('reasoning')
      break
  }

  const systemTokens = approximateTokens(input.systemPrompt ?? '')
  const userTokens = approximateTokens(input.userPrompt)
  const inputTokens = systemTokens + userTokens

  if (inputTokens > 100_000) {
    requiredCapabilities.add('long-context')
  }

  if (input.attachments?.some(a => a.kind === 'image')) {
    requiredCapabilities.add('vision')
  }

  const estimatedOutputTokens = Math.min(
    4_000,
    Math.max(200, Math.ceil(inputTokens * 0.3))
  )

  return {
    requiredCapabilities,
    estimatedInputTokens: inputTokens,
    estimatedOutputTokens,
    highStakes: input.highStakes ?? false,
  }
}

The classifier is deliberately conservative. It does not try to "guess" whether a prompt needs reasoning by analyzing the text — that path leads to a tiny self-hosted classifier model that you then have to maintain forever. Instead, the caller declares the task. Routing on declared task plus declared high-stakes flag is good enough to capture roughly 90% of the savings with 5% of the engineering effort.

If you cannot trust callers to pick a task accurately, run a cheap one-shot classification with Gemini 3.1 Flash-Lite at the edge of your API gateway and write the result as a header. That classifier call costs roughly $0.0001 per request — utterly invisible against the routing savings it unlocks.

Step 3: The Router

The router takes a RoutingRequirement and returns the cheapest model that satisfies it.

Create src/router.ts:

import { MODELS, type ModelSpec } from './registry'
import type { RoutingRequirement } from './classifier'

export interface RoutingDecision {
  primary: ModelSpec
  fallbacks: ModelSpec[]
  estimatedCostCents: number
  reason: string
}

function covers(model: ModelSpec, req: RoutingRequirement): boolean {
  for (const c of req.requiredCapabilities) {
    if (!model.capabilities.has(c)) return false
  }
  if (req.estimatedInputTokens > model.maxContextTokens) return false
  if (req.highStakes && model.tier === 'floor') return false
  return true
}

function estimatedCostCents(model: ModelSpec, req: RoutingRequirement): number {
  const inputDollars =
    (req.estimatedInputTokens / 1_000_000) * model.inputCostPerMillion
  const outputDollars =
    (req.estimatedOutputTokens / 1_000_000) * model.outputCostPerMillion
  return Math.round((inputDollars + outputDollars) * 100 * 1000) / 1000
}

const TIER_ORDER: Record<ModelSpec['tier'], number> = {
  floor: 0,
  mid: 1,
  frontier: 2,
}

export function route(req: RoutingRequirement): RoutingDecision {
  const candidates = MODELS.filter(m => covers(m, req))

  if (candidates.length === 0) {
    throw new Error(
      `No model in registry satisfies requirements: ${[
        ...req.requiredCapabilities,
      ].join(', ')}`
    )
  }

  // Sort by tier ascending (cheapest first), then by cost, then by eval score
  // descending as the tiebreaker.
  const sorted = [...candidates].sort((a, b) => {
    const tierDelta = TIER_ORDER[a.tier] - TIER_ORDER[b.tier]
    if (tierDelta !== 0) return tierDelta
    const costDelta = estimatedCostCents(a, req) - estimatedCostCents(b, req)
    if (costDelta !== 0) return costDelta
    return b.evalScore - a.evalScore
  })

  const primary = sorted[0]
  const fallbacks = sorted.slice(1).filter((m, i, arr) => {
    // Only keep one fallback per tier; skip duplicates.
    return arr.findIndex(other => other.tier === m.tier) === i
  })

  return {
    primary,
    fallbacks,
    estimatedCostCents: estimatedCostCents(primary, req),
    reason: `Cheapest model in tier '${primary.tier}' covering [${[
      ...req.requiredCapabilities,
    ].join(', ')}]`,
  }
}

Notice the fallback list is filtered to one model per higher tier. That is deliberate. If your floor model 429s, you do not want to silently retry against a second floor model that is going to 429 for the same reason (rate limits on small providers tend to be correlated across requests, not across providers). You want the next tier up, then the tier above that. Three retries, three tiers, then surface the error.

Step 4: Wiring Up Real Providers

The router decides which model to call. Provider-specific code does the actual HTTP. Keep them separate.

Create src/providers.ts:

import type { ModelSpec } from './registry'

export interface CompletionRequest {
  systemPrompt?: string
  userPrompt: string
  maxOutputTokens: number
}

export interface CompletionResponse {
  text: string
  inputTokens: number
  outputTokens: number
}

export class ProviderError extends Error {
  constructor(
    message: string,
    public readonly retryable: boolean,
    public readonly status?: number
  ) {
    super(message)
  }
}

type ProviderFn = (
  model: ModelSpec,
  req: CompletionRequest
) => Promise<CompletionResponse>

const providers: Record<ModelSpec['provider'], ProviderFn> = {
  anthropic: async (model, req) => {
    const response = await fetch('https://api.anthropic.com/v1/messages', {
      method: 'POST',
      headers: {
        'x-api-key': process.env.ANTHROPIC_API_KEY!,
        'anthropic-version': '2023-06-01',
        'content-type': 'application/json',
      },
      body: JSON.stringify({
        model: model.id,
        max_tokens: req.maxOutputTokens,
        system: req.systemPrompt,
        messages: [{ role: 'user', content: req.userPrompt }],
      }),
    })

    if (!response.ok) {
      throw new ProviderError(
        `Anthropic ${response.status}: ${await response.text()}`,
        response.status === 429 || response.status >= 500,
        response.status
      )
    }

    const json = (await response.json()) as {
      content: { text: string }[]
      usage: { input_tokens: number; output_tokens: number }
    }
    return {
      text: json.content.map(c => c.text).join(''),
      inputTokens: json.usage.input_tokens,
      outputTokens: json.usage.output_tokens,
    }
  },

  openai: async (model, req) => {
    const response = await fetch('https://api.openai.com/v1/responses', {
      method: 'POST',
      headers: {
        authorization: `Bearer ${process.env.OPENAI_API_KEY!}`,
        'content-type': 'application/json',
      },
      body: JSON.stringify({
        model: model.id,
        max_output_tokens: req.maxOutputTokens,
        input: [
          ...(req.systemPrompt
            ? [{ role: 'system', content: req.systemPrompt }]
            : []),
          { role: 'user', content: req.userPrompt },
        ],
      }),
    })

    if (!response.ok) {
      throw new ProviderError(
        `OpenAI ${response.status}: ${await response.text()}`,
        response.status === 429 || response.status >= 500,
        response.status
      )
    }

    const json = (await response.json()) as {
      output_text: string
      usage: { input_tokens: number; output_tokens: number }
    }
    return {
      text: json.output_text,
      inputTokens: json.usage.input_tokens,
      outputTokens: json.usage.output_tokens,
    }
  },

  google: async (model, req) => {
    const url =
      `https://generativelanguage.googleapis.com/v1beta/models/${model.id}:generateContent` +
      `?key=${process.env.GOOGLE_API_KEY}`
    const response = await fetch(url, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({
        systemInstruction: req.systemPrompt
          ? { parts: [{ text: req.systemPrompt }] }
          : undefined,
        contents: [{ role: 'user', parts: [{ text: req.userPrompt }] }],
        generationConfig: { maxOutputTokens: req.maxOutputTokens },
      }),
    })

    if (!response.ok) {
      throw new ProviderError(
        `Google ${response.status}: ${await response.text()}`,
        response.status === 429 || response.status >= 500,
        response.status
      )
    }

    type Part = { text: string }
    const json = (await response.json()) as {
      candidates: { content: { parts: Part[] } }[]
      usageMetadata: {
        promptTokenCount: number
        candidatesTokenCount: number
      }
    }
    const text = json.candidates[0].content.parts.map(p => p.text).join('')
    return {
      text,
      inputTokens: json.usageMetadata.promptTokenCount,
      outputTokens: json.usageMetadata.candidatesTokenCount,
    }
  },

  mistral: async () => {
    throw new ProviderError('Mistral not implemented in this tutorial', false)
  },
}

export async function callProvider(
  model: ModelSpec,
  req: CompletionRequest
): Promise<CompletionResponse> {
  return providers[model.provider](model, req)
}

This is uglier than the rest of the code on purpose. Provider APIs differ in small, annoying ways, and the only way to use multiple providers correctly is to translate each one at the boundary. Do not try to invent a "neutral" abstraction that pretends every API has the same shape — you will spend more time maintaining the translation layer than you save in routing.

Step 5: Putting It Together

Now the orchestrator. This is what your application code actually calls.

Create src/index.ts:

import { classify, type PromptInput } from './classifier'
import { route, type RoutingDecision } from './router'
import {
  callProvider,
  ProviderError,
  type CompletionResponse,
} from './providers'
import { logRouting } from './telemetry'

export interface RouterResult extends CompletionResponse {
  decision: RoutingDecision
  modelUsed: string
  actualCostCents: number
  fallbacksAttempted: number
}

export async function ask(input: PromptInput): Promise<RouterResult> {
  const requirement = classify(input)
  const decision = route(requirement)

  const ladder = [decision.primary, ...decision.fallbacks]
  let lastError: unknown

  for (let i = 0; i < ladder.length; i++) {
    const model = ladder[i]
    try {
      const completion = await callProvider(model, {
        systemPrompt: input.systemPrompt,
        userPrompt: input.userPrompt,
        maxOutputTokens: requirement.estimatedOutputTokens,
      })

      const actualCostCents =
        ((completion.inputTokens / 1_000_000) * model.inputCostPerMillion +
          (completion.outputTokens / 1_000_000) * model.outputCostPerMillion) *
        100

      const result: RouterResult = {
        ...completion,
        decision,
        modelUsed: model.id,
        actualCostCents: Math.round(actualCostCents * 1000) / 1000,
        fallbacksAttempted: i,
      }
      logRouting(input, result)
      return result
    } catch (error) {
      lastError = error
      if (error instanceof ProviderError && !error.retryable) {
        throw error
      }
    }
  }

  throw lastError
}

That is the whole router. Roughly 30 lines of orchestration code on top of a classifier and a registry. The pattern is dumb on purpose — there is no state, no learned policy, no LLM-as-a-judge in the hot path. Dumb routers survive contact with production traffic in a way that clever routers do not.

Step 6: Cost Telemetry

This is the most important step and the one most teams skip. If you cannot prove the router saved money, the router does not exist.

Create src/telemetry.ts:

import type { PromptInput } from './classifier'
import type { RouterResult } from './index'

export interface RoutingLogEntry {
  ts: string
  task: string
  modelUsed: string
  modelTier: string
  fallbacksAttempted: number
  estimatedCostCents: number
  actualCostCents: number
  inputTokens: number
  outputTokens: number
  // What the dumb baseline would have cost: every call to Opus 4.7.
  baselineCostCents: number
  savedCents: number
}

const OPUS_INPUT_PER_MILLION = 15.0
const OPUS_OUTPUT_PER_MILLION = 75.0

export function logRouting(input: PromptInput, result: RouterResult): void {
  const baselineCostCents =
    ((result.inputTokens / 1_000_000) * OPUS_INPUT_PER_MILLION +
      (result.outputTokens / 1_000_000) * OPUS_OUTPUT_PER_MILLION) *
    100

  const entry: RoutingLogEntry = {
    ts: new Date().toISOString(),
    task: input.task,
    modelUsed: result.modelUsed,
    modelTier: result.decision.primary.tier,
    fallbacksAttempted: result.fallbacksAttempted,
    estimatedCostCents: result.decision.estimatedCostCents,
    actualCostCents: result.actualCostCents,
    inputTokens: result.inputTokens,
    outputTokens: result.outputTokens,
    baselineCostCents: Math.round(baselineCostCents * 1000) / 1000,
    savedCents:
      Math.round((baselineCostCents - result.actualCostCents) * 1000) / 1000,
  }

  // In production: write to your metrics pipeline (DataDog, Honeycomb,
  // OpenTelemetry). Here: stdout JSON, parseable by anything.
  console.log(JSON.stringify(entry))
}

The trick is the baselineCostCents field. By logging what every request would have cost on Opus 4.7, you can answer "how much did the router save this month?" with a one-line aggregation over your logs. You do not need a A/B test, you do not need to keep the baseline model around — the baseline is computed for free from the same token counts the real call returned.

This is the spend curve from a representative deployment: a customer support tool serving roughly 60,000 LLM calls per week. Week 1 is the all-Opus baseline. Week 2 the router goes live. Spend drops 67% in week one, stabilizes around 75% savings by week four as the classifier dials in.

Step 7: The Replay Harness

Pricing changes. Providers ship new models. The router's behavior shifts underneath you and you do not know it shifted until the bill arrives. Pin behavior with a fixture-based test suite.

Create __tests__/router.test.ts:

import { describe, it, expect } from 'vitest'
import { classify, type PromptInput } from '../src/classifier'
import { route } from '../src/router'

interface Fixture {
  name: string
  input: PromptInput
  expectedTier: 'floor' | 'mid' | 'frontier'
  expectedModelId?: string
}

const fixtures: Fixture[] = [
  {
    name: 'simple email extraction → floor',
    input: {
      task: 'extract',
      userPrompt: 'Find the email address: contact me at foo@bar.com today.',
    },
    expectedTier: 'floor',
    expectedModelId: 'gemini-3.1-flash-lite',
  },
  {
    name: 'short summary → floor',
    input: {
      task: 'summarize',
      userPrompt: 'Summarize: ' + 'lorem ipsum '.repeat(500),
    },
    expectedTier: 'floor',
  },
  {
    name: 'multi-step reasoning → mid',
    input: {
      task: 'reason',
      userPrompt:
        'Given quarterly revenues 1.2M, 1.4M, 1.1M, 1.6M, what is the trend?',
    },
    expectedTier: 'mid',
  },
  {
    name: 'agent with tools, high stakes → frontier',
    input: {
      task: 'agent',
      userPrompt: 'Audit this 2000-line refactor and propose a migration plan',
      highStakes: true,
    },
    expectedTier: 'frontier',
  },
  {
    name: 'long-context summarization → mid, not frontier',
    input: {
      task: 'summarize',
      userPrompt: 'a'.repeat(800_000), // ~200k tokens
    },
    expectedTier: 'mid',
  },
]

describe('router', () => {
  for (const f of fixtures) {
    it(f.name, () => {
      const decision = route(classify(f.input))
      expect(decision.primary.tier).toBe(f.expectedTier)
      if (f.expectedModelId) {
        expect(decision.primary.id).toBe(f.expectedModelId)
      }
    })
  }
})

Run npx vitest. When a fixture starts producing a different decision, your build fails. That is the point. Either the price change is fine and you update the fixture, or it is a regression and you stop the merge. Either way the behavior is no longer silent.

What This Actually Costs To Run

Let me put numbers on the savings, because "60-85%" is too vague to make a build/buy decision.

That is real data — slightly anonymized — from a SaaS company running a support copilot. Jan was the all-Opus baseline. They deployed the router in late January. Monthly LLM bill went from $48,200 to $11,900 — a 75% reduction on traffic that grew 31% over the same window. Engineering time to build the router and ship it: roughly two weeks for one engineer.

Your numbers will not be exactly these. The shape will be similar. The specific mix of floor / mid / frontier depends on your prompt distribution.

Operations: What To Monitor

Once the router is in production, the dashboards you need are different from the ones you had with a single-model client. The single-model world tracks latency, errors, and tokens. The routed world adds three more signals, and each one will tell you something a single-model dashboard cannot.

The first is the tier distribution shift. Plot the share of traffic going to floor / mid / frontier over time, broken down by task. A healthy router keeps tier shares roughly stable for a stable workload. When the distribution drifts — say frontier share doubles from 8% to 16% over a week — something changed. Often it is a new high-stakes flag added by an upstream team. Often it is a provider quietly degrading the floor model and your fallback ladder is silently catching it. Either way, you want a chart, not a surprise.

The second is the fallback escalation rate. Every successful retry up the ladder is money you spent that you did not budget for. Alert when more than five percent of requests in any five-minute window required a fallback. That threshold is conservative; pick your own. The point is to make silent escalation loud. A floor provider that 429s 30% of the time is not actually free.

The third is the cost-per-task histogram. Group every request by task type, then chart the actual cost distribution. Outliers will jump out visually: a "summarize" task that spent eight cents is doing something unusual. Usually it is a prompt that ballooned in size because someone fed the model a 200,000-token document and asked for a summary. The router correctly escalated to a long-context model — and the price tag came with it. Knowing about it before the bill arrives is the difference between fixing the upstream caller and just paying.

A fourth signal that is harder to instrument but worth the effort: quality parity sampling. Pick one percent of routed traffic at random. Run the same prompt through the next tier up. Score both answers with your eval harness or, failing that, with a frontier-tier judge model. If the floor-tier answer scores within five points of the higher-tier answer, the router made the right call. If it scores ten points lower, you have a routing bug. Aggregate the parity rate weekly. If it dips below 90%, your classifier is letting prompts through that need the higher tier. Tune the high-stakes heuristic or add a new declared task.

None of these dashboards require expensive infrastructure. They are aggregations of the same RoutingLogEntry rows you are already writing. If you are using OpenTelemetry, every field maps to an attribute on a span. If you are using a structured-log SaaS, every field is already indexable. The work is in deciding what to alert on, not in collecting the data.

When Not to Route

Routing is not always the right architecture. Three cases where the dumb baseline beats the smart router:

You serve fewer than ten thousand requests per month. Two weeks of engineering time to save a few hundred dollars a month is not a deal. Use the frontier model, keep the codepath simple, and revisit when traffic grows.

Every request really does need the frontier. Some products — a legal research copilot, a senior-engineer pair programmer, a high-stakes medical triage tool — cannot accept floor-tier failure modes. Routing those is not worth the audit cost. Pay the frontier price, write it into the unit economics, and price the product accordingly.

Your prompts are within a narrow capability band. If 100% of your traffic is "extract structured JSON from this short document", you do not need a router. You need a hard-coded call to the cheapest model that can do that job. Routing is for portfolios of prompt shapes; constant-shape traffic skips the routing tax entirely.

Routing is for the messy middle: a meaningful prompt-shape distribution, a material monthly bill, and at least one engineer who can own the harness. Most production AI products fit that description by their second year.

Cost-Aware Engineering as a Cultural Shift

The hardest part of shipping a router is not the code. It is convincing everyone on your team that cost is a feature, not a regulatory burden. The default culture in AI engineering — for entirely understandable reasons — is "use the best model, measure quality, ignore the bill until it bites." That culture worked when the bill was a thousand dollars a month. It does not scale to six figures.

The router changes the conversation. Suddenly every team can see, per endpoint, what each call costs. Suddenly product managers can ask "what percentage of these calls are landing on Opus?" and get a number. Suddenly the right answer to "should we use a more expensive model for this feature?" is data instead of vibes. That cultural shift is worth more than the dollars saved on day one. It compounds quarter over quarter.

If you take one thing away from this tutorial, take this: ship the telemetry before you ship the routing. A team that can see its inference costs broken down by task already has the most expensive problem solved. The router is just the mechanism that turns visibility into savings.

Five Pitfalls I Have Hit

These are not theoretical. Each one cost me at least a day.

1. Trusting the eval score from a benchmark blog post. Use your own eval harness. The MMLU number on a vendor's announcement slide does not predict how the model handles your customers' typos. Build the harness first. I've written about why eval harnesses moved in-house — same logic applies here.

2. Routing on prompt length alone. A 50-token prompt asking for a multi-step audit is not a floor-tier prompt. A 100,000-token prompt asking for five-word categorization is. Length is one input, not the input.

3. Caching at the wrong layer. If you cache identical prompts at the router level, you skip routing entirely on cache hits — which sounds great until the cache returns a stale answer for a prompt that should now go to a different tier because pricing changed. Cache below the router, not above it.

4. Silent fallback escalation. The fallback ladder is a kindness when it catches transient 5xxs and a disaster when it silently runs every request at frontier price because your floor provider went down for two hours and nobody noticed. Alert on the rate of requests where fallbacksAttempted is non-zero, not just on errors.

5. Forgetting the routing call itself is not free. If you classify with an LLM (even Flash-Lite), every routed request now makes two API calls. That is fine if the classification call costs $0.0001 and the saved call would have cost $0.04. It is catastrophic if you accidentally route the classifier itself through the router. Yes, I have done this.

The bar chart shows the tradeoff for four classifier strategies. Pure length-based routing is cheap but inaccurate. Letting callers declare the task is almost as accurate as a full LLM classifier and meaningfully cheaper. A small cheap-classifier model in front of the router splits the difference. Using a frontier model as the classifier defeats the purpose: you pay more for classification than the routing saves.

Going Further

Three extensions worth building once the basic router is live:

Streaming. The interfaces above assume single-shot completions. Add a streaming path that proxies tokens from the provider through the router back to the caller. The router does not change — only callProvider does. Each provider's streaming format is different; encapsulate them at the provider boundary the same way you did the non-streaming ones.

Per-tenant policy. Some customers will be willing to pay for frontier defaults. Some will not. Add a tenantPolicy parameter that biases the router (e.g., cheapestUnlessFrontierRequested vs cheapestAcceptable vs alwaysFrontier) and let the policy live in customer config.

Eval feedback loop. Periodically replay a sample of production prompts against the next tier up. If the floor and mid answers agree on more than 95% of samples, your floor model is probably under-utilized — widen its capabilities. If they disagree more than 30%, your floor model is being asked to do things it cannot.

The first two are afternoon projects. The third is a quarterly investment but turns the router from a static cost optimizer into a system that adapts to your traffic.

Why This Pattern Holds Up

The router pattern bets on one specific market structure: a wide and persistent price gap between commodity inference and frontier inference. That gap exists today and is likely to widen rather than close. My prediction on GPT-4-class inference dropping below $0.50 per million tokens came in at $0.25 — half the predicted floor. Meanwhile frontier agentic pricing has held above the $4 input floor I forecast. Two trajectories, diverging, with no consolidation force in sight.

That is the market the router is built for. As long as floor prices keep falling and frontier prices keep rising, routing pays. The day they re-converge — five years out, maybe never — the router becomes redundant. Easy to delete; trivial to replace with a single client. The downside of building it is small. The downside of not building it grows linearly with your prompt volume.

Conclusion

Routing is the highest-leverage cost intervention available to any team running production LLM traffic in 2026. A two-week build typically saves 60-85% of inference spend on workloads of any reasonable size. The code is short, the math is honest, and the failure modes are visible if you log them.

The companion repo is at CrashBytes/ByteSizedExamples/cost-aware-multi-model-router-typescript. Clone it, run npm install && npm test to see all 30 tests pass offline, then npm run demo to watch tasks route through the ladder — or point it at your provider keys and start routing.

Further Reading

• The inference price floor at $0.25/M and the bifurcating fast/frontier tiers — the May 2026 pricing snapshot this tutorial is calibrated to.
• The agentic foundation model reset — how frontier pricing doubled and stayed there.
• Why engineering orgs built their own eval harnesses — the harness is the prerequisite to trusting the router's tier choices.