The Next Model Is a System: Building the Mixture-of-Models Era

The next model architecture will not fit inside one checkpoint.

vLLM Semantic Router is now building Mixture-of-Models, a higher-level model architecture that turns a portfolio of independent models into one trainable, evaluable, and deployable model.

In less than a year, the project has grown to 5,000 stars, 150+ contributors, and more than 300,000 cumulative downloads across our Hugging Face model family. We shipped three major releases—Iris, Athena, and Themis—and watched a small experiment in semantic model selection turn into a serious open-source system for multi-model inference.

Those milestones close our first chapter. More importantly, they give us the technical foundation and the community for what comes next:

vLLM Semantic Router is becoming the training, evaluation, and inference engine for Mixture-of-Models.

Mixture-of-Models, or MoM, has been part of our system design from day 0. Until now, most of our public work has focused on the mechanism that makes it possible: routing. The next chapter moves the abstraction boundary higher. We want a complete model system—made from many models, policies, preferences, and execution paths—to be trained, evaluated, exported, imported, and run as one model.

If vLLM made a model fast and easy to serve, vLLM Semantic Router should make many models feel like one.

Figure 1: A Mixture-of-Models turns a heterogeneous model portfolio into one model experience.

How vLLM-SR Got Here

The first vLLM Semantic Router post started with a narrow, practical problem. Simple requests did not need the same reasoning budget as hard ones, so a lightweight classifier chose between fast and reasoning paths. The early router classified prompts into a fixed set of domains and helped vLLM spend inference compute more selectively.

That design was useful, but production traffic quickly broke its assumptions. A request is not defined by domain alone. It may also carry privacy constraints, safety risk, long context, language, modality, tool state, user preference, latency pressure, and authorization boundaries. A model endpoint can be cheap but overloaded, capable but remote, private but too small, or excellent for one turn and unsafe to switch into halfway through an agent session.

That gap forced the architecture to expand. We first rebuilt the classifier layer around modular model support, shared LoRA computation, Rust/Candle inference, and Go integration. We then replaced fixed classification with a Signal–Decision architecture, separating observed evidence from policy and execution. That separation became the spine of the next three releases.

Milestone	What changed
Initial release	Intent-aware selection between fast and reasoning paths
v0.1 Iris	Signals, decisions, and route-scoped plugins replaced fixed classification
v0.2 Athena	Model selection, memory, RAG, long context, multimodality, and a new router model stack expanded routing into an inference control system
v0.3 Themis	Stateful routing, projections, replay, protocol support, session continuity, and one production configuration contract made the system operable
Fusion and Micro-Agent	The router began choosing collaboration patterns, not only individual models

Figure 2: Each stage changed the unit of control: model, decision, system, session, and finally the complete model lifecycle.

Iris made routing composable. Signals for domain, keywords, embeddings, factuality, feedback, and preference fed explicit decisions; safety, PII protection, semantic caching, hallucination detection, and tool selection became route-scoped behavior. Iris also introduced the MoM model family and described vLLM-SR as “System Level Intelligence for Mixture-of-Models.”

Athena expanded the scope. It refreshed the multilingual and multimodal model stack, made model-selection algorithms first-class, brought memory and RAG into the runtime, accelerated long-context classifiers on AMD ROCm, and turned the dashboard into an operating surface. The project was no longer just a classifier in front of vLLM; it was becoming the control system around multi-model inference.

Themis turned that broader system into an operable contract:

Signals become projections. Projections feed decisions. Decisions choose algorithms. Algorithms select models.

Themis added session-aware agentic routing, replayable traces, stronger OpenAI and Anthropic protocol support, an operator console, and broader runtime paths across AMD ROCm, NVIDIA CUDA, Intel OpenVINO, and CPU environments. It made an important promise to operators: a route should be explainable. You should be able to see what evidence fired, which policy matched, which algorithm ran, which physical model served the request, and why.

From Signal–Decision to Workload–Router–Pool

The releases built the runtime. Two project papers explained the architecture behind it.

The white paper, Signal Driven Decision Routing for Mixture-of-Modality Models, formalized the separation between neural evidence and symbolic policy. It describes routing in two regimes. In the first, signal extraction reduces uncertainty about which model or path fits a request. Fast heuristics and learned classifiers turn raw prompts, context, identity, safety, and modality into a structured signal vector. In the second, a Boolean decision engine composes those signals into an auditable policy. At the time of the paper, the system covered thirteen signal types and thirteen model-selection algorithms, with per-decision plugin chains for caching, RAG, memory, safety, provider handling, and response validation.

The typed neural-symbolic DSL is an important part of that design. It acts as the instruction set for the routing engine: parse policy into a Boolean expression tree, validate syntax and references, and compile the same intent into deployable configuration. Neural models remain good at extracting uncertain evidence; symbolic policy remains reviewable, testable, and suitable for infrastructure control.

The vision paper, The Workload–Router–Pool Architecture for LLM Inference Optimization, widened the frame. It argues that three variables have to be designed together:

• Workload: chat or agent, single-turn or multi-turn, warm or cold, prefill-heavy or decode-heavy
• Router: static semantic policy, online feedback or bandit adaptation, RL-based selection, and quality-aware cascades
• Pool: homogeneous or heterogeneous accelerators, prefill/decode topology, model placement, and KV-cache management

Those variables cannot be optimized independently. Workload shape changes which routing policy works; routing policy changes the required pool size and topology; pool state changes which route is efficient. Safety and privacy cut across all three dimensions, while cost, quality, latency, and energy define the optimization frontier. The paper maps the project’s research into a 3 × 3 WRP matrix and identifies twenty-one open directions where those dimensions still need to meet.

Figure 3: The white paper defines the programmable routing engine; the vision paper connects it to workload and physical pool design.

The two papers supplied different pieces of the same architecture. The white paper made routing decisions programmable; the vision paper tied those decisions to workload and hardware. Mixture-of-Models puts both under one model contract.

Meanwhile, the runtime was already moving beyond single-model selection. Fusion, ReMoM, Confidence, Ratings, and bounded Workflows let one request invoke a controlled collaboration among models. As the Micro-Agent work showed, a client can call one model name while the serving layer selects a recipe, fans out to workers, verifies or synthesizes their results, and returns one ordinary response.

That shift marks the boundary between the first chapter and the next.

First chapter	New chapter
Route a request	Build a model system
Choose a model or capability path	Train, evaluate, and execute the whole MoM
Configure runtime policy	Package a portable, versioned model artifact
Optimize a routing decision	Optimize system intelligence across quality, cost, latency, safety, and energy
Hide backend choice behind one API	Make the complete multi-model system behave like one model

Routing remains fundamental. It is how a Mixture-of-Models allocates work, applies policy, and coordinates its parts. But routing is the mechanism. The model system is the product.

Why the Model Boundary Has to Move

Today’s AI stack is already fragmented. Its components may be individually powerful, but the system boundary between them is still missing.

Models are fragmented. Closed frontier models, open general models, domain experts, compact local models, verifiers, embeddings, and multimodal models will coexist. None wins across quality, cost, latency, trust, privacy, modality, and domain fit at the same time.

Compute is fragmented. New and old GPUs, CPUs, specialized accelerators, edge devices, cloud capacity, and private clusters have different memory limits, kernels, availability, prices, and energy curves. Model choice and compute placement are becoming the same decision.

Location is fragmented. Inference now spans cloud, data center, and edge. Some data must stay local. Some workloads benefit from an on-demand cloud expert. Others must remain inside a regulated boundary even when a remote model would score better on a benchmark.

Preference is fragmented. There is no universal “best.” Products and users make different tradeoffs among accuracy, latency, price, privacy, hallucination tolerance, safety, style, and multimodality. Those preferences should shape execution directly.

Before MoM, every product has to assemble those pieces for itself. Model selection sits in SDK wrappers, provider rules live in gateways, safety and privacy checks are scattered across middleware, and agent graphs carry their own fallback and collaboration logic. Two applications using the same model portfolio can end up with different behavior, different traces, and no shared way to evaluate the complete system.

Figure 4: Before MoM, fragmented intelligence becomes application-side routing glue.

The durable bottleneck is therefore not model access. It is intelligent allocation: deciding which intelligence should run, where it should run, how much of it to spend, and whether several models should work together.

This is also an energy problem. Hardware and inference engines improve the supply side by producing more tokens per watt per dollar. MoM addresses the demand side: which semantic work deserves those tokens at all? Sending every request to the largest model wastes capacity; forcing every request onto the cheapest model wastes capability. A mature system has to know the difference.

MoM does not make this heterogeneous stack homogeneous. It moves the complexity behind one model boundary. The engine owns allocation, collaboration, constraints, fallbacks, and traces; the application chooses one versioned model identity and receives one response. The physical realization can still span open and closed models, cloud and edge, and different accelerator generations without exposing that machinery to every caller.

Figure 5: With MoM, the same fragmented resources become the internal realization of one model.

Once that boundary exists, training, evaluation, and inference can operate on the same system.

What We Mean by Mixture-of-Models

A Mixture-of-Models is a versioned composite model whose engine realizes each request through a preference-conditioned, resource-bounded path across independent models and operators. It is presented to the user through one model interface and returns one attributable result.

MoM is not another name for a gateway with several upstreams. A gateway can forward traffic without owning the quality of the resulting system. An MoM has an objective, an evaluation contract, a reproducible composition, and a runtime responsible for executing that composition.

MoM is also different from Mixture-of-Experts. MoE routes tokens among internal experts during one model’s forward pass. MoM coordinates independent models at the system level. Its components may use different architectures, owners, licenses, modalities, protocols, context windows, and hardware. An MoE checkpoint can itself be one component inside an MoM.

	Conventional model	Mixture-of-Models
Unit of intelligence	One checkpoint	A governed system of models
Specialization	Primarily encoded in weights	Composed across independent specialists
Execution	One generation path	Selection, cascade, verification, fusion, or workflow
Optimization target	One model’s quality and efficiency	The system frontier across quality, cost, latency, safety, privacy, and energy
Deployment boundary	One runtime	Cloud, data center, and edge
User contract	One model identity	One model identity

Figure 6: Selection is one MoM topology. Cascades, parallel fusion, and bounded workflows share the same model boundary.

A portable MoM therefore needs more than weights and configuration: it also needs a component manifest, capability metadata, routing and collaboration recipes, policies, preferences, evaluation suites, runtime constraints, provenance, and version history.

Open checkpoints can travel with the artifact. Closed models can be preserved as authenticated external references with explicit capability and policy contracts. Exporting an MoM does not make a proprietary checkpoint portable; it makes the model system reproducible.

Preference as a model contract

This abstraction becomes useful when it is as concrete as a model name. The draft Mixture-of-Models Bundle proposal and position paper illustrate one family with several published operating points:

Model identity	Contract
vllm-sr/mom-v1-flash	Minimize expected latency
vllm-sr/mom-v1-light	Minimize cost above a quality floor
vllm-sr/mom-v1-ultra	Maximize quality within a declared budget
vllm-sr/mom-v1-halu	Require grounding checks and fail-closed fallback
vllm-sr/mom-v1-secu	Enforce jailbreak and PII policy before execution

These names make preference part of the model contract. flash is trained and evaluated on a latency–quality frontier; ultra can spend a declared budget to pursue higher quality; halu and secu make grounding and security gates part of the model itself. When the family advances, mom-v2-ultra can introduce a new policy or topology without silently changing mom-v1-ultra. The application still chooses a model. vLLM-SR owns the complexity behind it.

A request can still tune a published tradeoff within the model’s declared range. For example, ultra may accept a small latency bias while remaining accuracy-first. It cannot quietly become light while keeping the ultra identity. No soft preference may override a privacy, residency, authorization, or safety constraint. The engine decides what is admissible before it decides what is optimal.

To an application, the full system remains an ordinary model call:

{
  "model": "vllm-sr/mom-v1-ultra",
  "messages": [
    {"role": "user", "content": "Review this design and identify its weakest assumption."}
  ]
}

Internally, that identity may choose one model, escalate through a cascade, compare parallel answers, require a grounding check, or run a bounded workflow. Externally, it keeps one interface, one version, and one response.

Figure 7: Preferences are published as bounded, versioned model contracts—not hidden application-side routing presets.

The architecture we are building has four planes with clear ownership:

Plane	What it owns	Foundation already in vLLM-SR	Next step
Artifact	Components, capabilities, objectives, policy, eval contract, provenance	Canonical config, model references, DSL, versioned policy	Portable MoM import/export specification
Learning	Router-owned models, preferences, outcomes, recipe improvement	Training stack, Router Learning, replay, outcome APIs	Joint training and system-level release gates
Execution	Signals, projections, decisions, selectors, loopers, plugins	Signal–Decision runtime, Fusion, ReMoM, Workflows, safety and memory	One lifecycle-aware MoM engine
Physical	Providers, model pools, accelerators, locality, cache and energy state	vLLM backends, cloud providers, ROCm, CUDA, OpenVINO, CPU	Portable placement across cloud, data center, edge, and local devices

Figure 8: A complete MoM spans four planes: artifact, learning, execution, and physical realization.

The portable artifact is only the first boundary. A deployment has to map logical requirements onto the models and machines available in one environment. The proposal separates that lifecycle into four objects:

1. The bundle fixes the model interface, graph, policies, contracts, behavior variant, bounds, and immutable semantic assets.
2. The binding maps logical components to eligible deployments without changing the model’s decision semantics.
3. The resolution lock freezes the exact constituent revisions, runtimes, images, accelerators, and provider observations used for a realization.
4. The run record attributes every decision, call, constraint check, cost, and outcome to the bundle, binding, and lock that produced it.

This separation is what makes portability honest. The same mom-v1-ultra can bind to a ROCm cluster, a CUDA fleet, a private CPU or NPU edge node, or a hybrid of local and managed models while preserving its logical contract. It does not promise identical outputs from opaque providers. It preserves the model’s control semantics, makes substitutions explicit, and gives serving and evaluation the same resolved system to execute.

vLLM-SR as the MoM Engine

Training, evaluation, and inference have to share one system contract. Otherwise routing research stays in notebooks, evaluation measures a different system from the one deployed, and production behavior drifts away from both.

Training allocation, not only weights

MoM training includes the models owned by the router—embeddings, signal encoders, preference models, safety models, and selectors—but it goes further. The system must learn which model or collaboration pattern performs best for a workload under a given budget and environment. It must learn stopping rules for cascades, judge and synthesis policies for panels, session-switch economics for agents, and preference-aware tradeoffs. Because constituent models may be independent or closed, progress does not require gradients through every model: policies, thresholds, model pools, prompts, contracts, and execution topology can all be optimized from traces and outcomes.

The optimization target is a frontier, not one accuracy number. Quality, latency, cost, safety, privacy, reliability, locality, and energy all matter. Replay and outcome data connect production experience back to offline training without allowing the hot path to rewrite policy silently.

Evaluating the MoM as one model

If users invoke one model identity, evaluation should score that identity end to end. Backend benchmark scores are inputs, not the final answer.

MoM evaluation needs to measure routing regret, collaboration gain, failure recovery, session continuity, tail latency, cost, privacy and safety outcomes, and tokens per watt. It should test provider failures, device loss, model disagreement, workload drift, and preference changes. Just as importantly, it must evaluate the declared operating point: flash on its latency–quality frontier, light against its quality floor, and ultra within its resource budget. The result should be a versioned scorecard that serves both as training evidence and as a release gate.

The scientific test is stricter than asking whether more model calls improve a benchmark. Under matched active compute, can a conditional system exploit complementary model strengths and failure modes better than the best fixed model? Without that control, MoM can hide brute-force scaling behind a clever graph. Our evaluations must therefore report active calls, tokens, cost, latency, and energy alongside quality—and publish the conditions where composition does not help.

Executing intelligence at inference time

At inference time, the engine decides whether one model is enough. It may choose a local specialist, preserve a warm session on the current model, escalate through a confidence cascade, require retrieval or verification, run a Fusion panel, or execute a bounded workflow. The runtime owns the budget, topology, fallback policy, trace, and response contract.

The application should not have to reconstruct that machinery. It should make a normal model call.

Figure 9: MoM is a closed lifecycle: train the allocation policy, evaluate the full system, execute it, and turn outcomes into the next validated version.

One Model That Can Move

Our target is a complete MoM that can be built, exported, imported, versioned, evaluated, deployed, and invoked as a unified model. The proposal deliberately puts that model lifecycle above today’s runtime configuration: author a logical specification, compile it into a typed intermediate representation, package an immutable bundle, bind it to an environment, resolve and lock the concrete deployment, then use that same identity for serving and evaluation.

The same artifact should run across a developer machine, a private cluster, a cloud fleet, and an edge environment while preserving its model identity and behavioral contract. Its physical realization can change with the environment: a logical specialist may resolve to an admissible local checkpoint or managed endpoint, and one accelerator runtime may be replaced by another. If a remote expert is unavailable behind a privacy boundary, the engine must follow a fallback or abstention path already declared by the model. The binding may not silently rewrite the graph, relax a guard, or turn a panel into a cascade; those changes require a new model version.

“Run on any hardware” is an architectural requirement, not a claim that every component is portable today. The project already has concrete paths across ROCm, CUDA, OpenVINO, and CPU execution. The next step is to make hardware capability and placement part of the MoM contract, so the engine can map the same model system onto what is actually available.

The standard for the user experience is simple:

One model identity. Many models. Any hardware.

Figure 10: One logical model identity can be realized across developer, data-center, cloud, and edge hardware.

If the application needs to know which provider owns every submodel, which device runs it, or which fallback graph to execute, the abstraction has leaked.

What Changes Now

The next stage of vLLM Semantic Router will concentrate on four connected areas:

1. Define a portable MoM specification. Represent components, capabilities, objectives, policies, preferences, evaluation contracts, runtime constraints, and finite execution semantics as one versioned artifact.
2. Close the training–evaluation–inference loop. Use evaluation and replay to improve router-owned models and recipes, validate changes against the system frontier, and deploy them through reviewable and rollback-safe releases.
3. Build a heterogeneous runtime. Map the same MoM across cloud, data center, and edge, using hardware, locality, energy, and data boundaries as execution inputs.
4. Keep the model interface boring. Import, export, deploy, and invoke an MoM with the same ease expected from a single model. Complexity belongs inside the engine.

Figure 11: Four connected workstreams turn Mixture-of-Models from an execution pattern into the next model architecture.

This is not a roadmap for a more elaborate router. It is a research program for how independent models should specialize, compete, verify, and collaborate; how we should measure the intelligence of the complete system; and how one model contract can survive across devices and environments. That is why we describe the mission as:

Advancing the science of intelligence across models, devices, and environments.

Across models, we want to learn when composition produces capability beyond one checkpoint. Across devices, placement, availability, and energy become part of the intelligence problem. Across environments, the same model contract should carry from edge to cloud and from research to production.

Build It With Us

Five thousand stars is not a finish line. It is evidence that this problem matters to more people than the original team.

Iris, Athena, and Themis were built in the open by people who trained models, opened issues, reviewed pull requests, added backends, tested releases, wrote documentation, shared benchmarks, and forced us to revise assumptions. The next chapter needs that same breadth: model and evaluation researchers, systems engineers, hardware teams, model builders, and operators with real workloads. We want collaborators working on learned allocation, preference optimization, model cooperation, energy-aware inference, portable artifacts, open evaluation, and heterogeneous runtimes.

Acknowledgments

vLLM-SR has always been shaped by people working across engineering, research, and the wider ecosystem. We are grateful to project leads Xunzhuo Liu (AMD) and Huamin Chen (Microsoft), and to Steering Committee members Bowei He, Yankai Chen, Fuyuan Lyu, and Steve Liu, for stewarding its technical and research direction. That work has grown through close collaboration with Andy Luo and Haichen Zhang from the AMD AIG team on ROCm enablement, router-model training, and open MoM experimentation.

We are equally grateful to the project’s full committer community, past and present: FAUST, David Shrader, Yang Wu, Ramakrishnan Sathyavageeswaran, Kuntai Wu, Aayush Saini, siloteemu, Chen Wang, Yue Zhu, Senan Zedan, Yossi Ovadia, Samzong Lu, Liav Weiss, Asaad Balum, Yehudit, Noa Limoy, Marina Koushnir, Jared Wen, Abdallah Samara, Hen Schwartz, Srinivas A, Yang Zhu, Jintao Zhang, yuluo-yx, cryo, Bishen Yu, Zhijie Wang, Hao Wu, and Qiping Pan. Their code, reviews, testing, documentation, and stewardship have carried the project from one release to the next.

At this milestone, the project stands at 1,733 commits and 150+ contributors. We are grateful to collaborators at MBZUAI, McGill University, Mila, and Rice University, and to the broader vLLM, AMD, Intel, Meta, Red Hat, Microsoft, Google, IBM, NVIDIA, Hugging Face, NASA, Nutanix, DaoCloud, and open-source communities. This milestone belongs to everyone who helped turn an early router into a real system.

Figure 12: Building the MoM engine is an open systems problem that needs the full model and infrastructure community.

Join us on GitHub, explore the documentation, try the MoM model family, and meet the community in the #semantic-router channel on vLLM Slack.

The first chapter taught infrastructure how to choose a model.

The next chapter is about making models, devices, and environments work as one model system.

Welcome to the Mixture-of-Models era.