Multi-Model LLM Routing That Cuts Inference Costs Without Losing Accuracy

A routing layer that dispatches queries across a small portfolio of privately hosted open-weight models, rather than a single flagship running every call, is the shortest path to lower inference cost at the accuracy a regulated workflow can defend. Published cascading and router architectures have cut costs by substantial margins at matched output quality, and the operational details that make those numbers survive a production SLA are what the rest of this piece lays out. The dispatcher itself becomes a compliance surface, which is the twist that changes how the architecture is governed.

The Three-Tier Portfolio Does the Work

A workable portfolio for enterprise traffic is three tiers: a small generalist such as Llama 3.1 8B, a mid-sized specialist such as Qwen 2.5 32B or Mixtral 8x22B, and a reasoning-grade model such as Llama 3.1 70B or 405B when the workload justifies the GPU footprint. The tiers do different jobs. The 8B handles classification, formatting, routing of its own kind, short summarization, and routine extraction at hundreds of tokens per second on a single H100 partition. The 32B tier absorbs retrieval-augmented generation against policy documents, SQL translation, and most drafting work. The 70B tier is held in reserve for multi-step reasoning, long-context synthesis, and anything where a wrong answer costs more than a minute of GPU time.

The portfolio shape matters more than any individual model choice. The serving stack and the quantization scheme determine whether tiers can live together on shared GPUs without contention, and the trade-off between vLLM, TensorRT-LLM and SGLang is where that decision gets made in practice.

How the Classifier Decides Which Model Answers

The dispatcher is a small model or a trained classifier that reads the prompt and emits a predicted tier plus a confidence score. Three implementations have earned production hours in the field:

1. Embedding similarity against pre-defined route examples, as in Aurelio's Semantic Router, which runs in milliseconds and keeps an LLM call off the hot path for simple classification.
2. A trained preference router such as Not Diamond's notdiamond-0001, which the vendor reports outperforms GPT-4 by a factor of 1.51x when used purely as a router.
3. A cascading architecture derived from FrugalGPT, which has shown matched quality against the strongest single model at up to 98% lower cost by escalating only when a lower tier's self-assessed confidence falls below a learned threshold.

Confidence thresholds should be set per task class, not globally. An entity-extraction call that fails on the 8B escalates to the 32B; a contract-clause rewrite starts at the 32B and falls through to the 70B when a citation check fails. Thresholds are hyperparameters, not constants, and they drift as traffic mix changes.

The Cost Math Per Million Tokens

Self-hosted inference on H100-class hardware moves each tier into its own cost band. The 8B tier runs for pennies per million tokens, the 32B tier for a small fraction of a dollar, and the 70B tier for one to three times the cost of the 32B depending on batch size and quantization. Those bands only earn their keep at steady use, since a dormant GPU costs the same as a saturated one. The arithmetic behind that threshold is in the published analysis of self-hosting against an API contract.

Prompt caching compounds the savings. Anthropic's published figures for cached-read pricing, which bill reads at ten percent of the base input-token rate and writes at a twenty-five percent premium, are the clearest public benchmark for what a well-tuned prefix cache is worth; open-source analogs give comparable economics on private hardware. The effect on long-context retrieval prompts is close to linear: a twenty-thousand-token system prompt and retrieval block, billed at full rate on every call, is the single largest line item in most pilots.

Where the Savings Compound

The second-order gains come from the serving layer. LMCache paired with vLLM reports time-to-first-token improvements of 3 to 10 times by reusing KV caches across engines and GPUs, which turns the second call on a shared prefix into a near-free event. Cache-aware query routing, pinning related requests to the engine that already holds their KV blocks, is the detail that pulls batched throughput above the published headline numbers.

Continuous batching, speculative decoding with a small draft model, and FP8 or INT8 weight quantization are the other three levers. Each is independent of routing but multiplies against it. The practical tuning order is routing first, to reduce the number of large-model calls, then caching, to cut the cost of the ones that remain, then quantization, to shrink the time each remaining call occupies on the GPU.

Why Each Routing Decision Is a Regulated Event

Under HIPAA, GLBA, CMMC, SOC 2, and most EU AI Act obligations, the system's answer is the regulated artifact. Under a routing architecture, so is the path to that answer. Every dispatcher decision writes a row to the audit log: prompt hash, selected tier, confidence score, policy applied, token counts, latency, and the identity of the operator service that issued the request. That row is what a compliance officer reads during a review, and what a defense counsel points to when asked why a particular output came from an 8B model rather than a 70B.

Governance guardrails sit around the dispatcher, not inside each model. A trained router that reroutes contract-review traffic to the small tier during a budget crunch is a material change to the risk profile, and it should surface the same way a code deploy does. Teams that treat the routing layer as part of their security audit and model review discipline catch these drifts before a regulator does.

What Shifts in the Next Twelve Months

The change most likely to reshape this decision over the next year is the compression of the open-weight tier structure itself. New 7B to 14B models are already approaching today's 70B class in reasoning, which flattens the portfolio and sharpens the question of when any call should escalate at all. That compresses the savings available from cascading but widens the quality floor, and the net direction is one a regulated operator should welcome. Teams investing now in a trained router, a defensible audit schema, and a custom LLM deployment plan that assumes the portfolio will change twice a year will keep the economics working as the models underneath shift.