How We Accidentally Trained the World's #1 Model Router

By Sqwish Labs · 2026-07-14

tl;dr We ask a small language model to inspect what it takes to answer each request well, then stop it before it generates a word and read the hidden state it has computed. The router combines that signal with data about each model's strengths to predict the accuracy and cost of each model at each reasoning budget, then picks the combination that best fits the current tradeoff. We would not have reached this result without synthetic training data, but generating synthetic data that actually improved the router was really, really hard.

We built this model router as an internal test of one part of LoopOne, then submitted it to RouterArena to see whether our ideas held up. Our submission took the #1 spot on the benchmark leaderboard. Here is the story of how we built it, with a live router you can try as you read.

We are building LoopOne to learn from an agent’s production traces and outcomes, then improve the decisions across its harness as it runs. For a whole agent, that learning problem is messy: decisions interact, and feedback can arrive late or never. Routing gave us one clean decision with immediate feedback: choose a model and reasoning budget, then observe accuracy and cost. RouterArena supplied 8,400 public requests for testing that loop. If we could not learn there, we were not ready for interacting decisions and missing or delayed outcomes.

A simple lookup table has no business being this good

Before calling the router useful, we wanted it to beat the choices a competent builder could make without one: first the best single model, then a lookup table built from historical evals. Beyond both lies the per-request best case, available only in hindsight.

The diamonds are the individual models in our pool. The fixed-model baseline picks one and sends it every request; the dotted envelope traces the best fixed-model choices across the cost range. On LLMRouterBench (Li et al. 2026)^[2], that “send everything to GPT-5” baseline beat three of six cost-aware routers, including the version of OpenRouter’s auto router evaluated in the paper. A routing layer that cannot beat the strongest fixed model has not earned its keep. No shade intended towards Microsoft Azure’s router submission in Figure 1, of course.

Gemini 3.1 Flash-Lite is the most accurate fixed model in our pool. Our submitted router, marked by the Sqwish star, is more accurate at roughly one-sixth the cost.

Then the family lookup nearly stopped us. An AI product builder usually knows which part of a workflow emitted a request before the router sees it. Code review, invoice reconciliation, support triage: each can get a model chosen from historical evals, then frozen into a lookup table. We call that the family lookup. It is a switch statement with a decent eval behind it, simple to build and stubbornly difficult to improve on. If a learned router cannot beat the lookup, it has spent machine learning to rediscover a decision the builder already knew. Ours did, after far more work than the fixed-model result required. That was the first result that felt non-trivial.

The oracle curve is not a deployable baseline. It is per-request hindsight: run every available model, wait for the outcomes, and keep the best one for each request. A live router has to choose before those outcomes exist, so the curve marks the theoretical ceiling rather than a policy anyone can serve.

Now look at the empty space. Our star clears the family lookup, but no learned router, ours included, comes close to the oracle. The family lookup tops out at 75.42% for $0.79 per thousand requests. Our submission reaches 76.40% for $0.18; hindsight reaches 82.79% at $0.23. The model pool contains much more complementarity than any router can yet exploit. We were mostly learning coarse topic and response-format signals, not the subtle task-specific structure where the remaining value lives. That gap became the problem we had to explain.

Maybe routing is like weather forecasting

One possibility was that a router looking only at the opening request could never predict everything that happens next.

LLM rollouts are stochastic, and they can also behave chaotically. Repeated inference can differ, even at zero temperature^[20], but that noise is the manageable part: run repeats and estimate the distribution. The harder problem is sensitivity. A small change to a prompt or an early token can send generation down a very different path. A tool call can reveal information that was absent from the request; a cheap model can get lucky, while an expensive one overthinks the easy case. For an agent, the clue that should determine which model to use may appear halfway through the trajectory. SWE-Router^[3] formalizes this for software agents: the initial issue description leaves some routing errors unavoidable, while a partial trajectory makes more of them predictable.

Some boundaries between easy and hard requests may simply be too irregular to learn reliably from the opening request alone.

A bad map can also make a predictable system look chaotic. Perhaps the routing boundary looked fractal because we had drawn it in the wrong coordinates.

Embeddings answer a different question

If the map is wrong, the representation underneath it is the obvious suspect. Most prompt routers begin with semantic embeddings or lexical features. We did too. Embeddings are inexpensive and excellent at grouping text by subject, but swapping among the strongest embedding models we could find barely changed our router.

Subject similarity and model difficulty do not have the same geometry. Two Python questions may sit next to each other in embedding space even when one asks for a library name and the other requires tracking a dozen interacting constraints.

A better retrieval representation can still be the wrong representation for model routing.

Just prompt it!

So semantic embeddings had stalled. The next idea sounded almost too obvious to count: ask a small language model what another language model would need to do to answer the request well.

The instruction steers the model’s prefill computation towards analysing the request. We stop before generation, keep the resulting hidden state, and combine it with compact NLP features. The resulting representation performed far better than our semantic-embedding variants: the structural features handled obvious cues, while the activations carried the routing signal those embeddings missed.

MiniCPM5-1B is the small language model we use to encode each request. When we compared hidden-state readouts from different depths, their ability to predict the best model family peaked just after the network’s midpoint and then fell toward the output. This agrees with Skean et al.^[4] on stronger intermediate representations and with Varshney et al.^[5] on routing from prefill activations.

For our engineering team’s continuing fight with latency, the peak was good news. We can stop the encoder at that layer and remove the later blocks from the serving path without losing routing signal in this experiment. Deleting part of a model is an unusually satisfying optimisation.

At almost the same time, Anthropic used its J-lens to report a shared representational “workspace” in Claude’s middle layers^[6], the same region where the routing signal peaks in Figure 3. In plain English, concepts arriving from different parts of a prompt appeared there in a common form: they could influence later reasoning and eventually be put into words. Global workspace theories associate that kind of information sharing with consciousness. Anthropic stopped short of declaring Claude conscious, but the surrounding internet was less restrained.

Since our routing signal had chosen roughly the same part of a much smaller network, we briefly wondered whether our ~1B model router had acquired a soul too. If so, it is narrowly scoped: it wants to maximise the return on your token spend by choosing the right model and reasoning budget.

Metaphysics aside, the J-lens offers something immediately useful: a way to inspect why the router chose one model over another. It is a change of coordinates: the model’s context-averaged Jacobian maps an intermediate activation into final-layer coordinates, where the ordinary unembedding turns it into a ranked vocabulary readout. Our router makes decisions from those same intermediate activations. Applied there, the lens could give us readable hypotheses about what shaped a routing decision: constraint, translation, multi-step reasoning, or something stranger.

We are saving the J-lens experiments for the next post.

A new model release should not reset the router

The prompted hidden state gives us a routing-friendly representation of the query. To choose a model, the router also needs to know what each model in the pool is good at.

A classifier-style router gives each model in the pool its own output class. When a new model arrives, that class has no history, so the router must be retrained before it can make a serious comparison.

To avoid rebuilding the router whenever the model list changes, we represent each candidate model with a capability fingerprint rather than a fixed output class. For public models, we build the fingerprint from domain benchmark scores, latency measurements and pricing data reported by independent evaluators such as Artificial Analysis^[7]. For private and fine-tuned models, we produce the equivalent profile with our own calibration suite. The existing predictor can score a new model immediately, then refine that estimate from the agent’s own outcomes as they arrive. This is the continual-learning pattern we want throughout LoopOne: start from general evidence, then update from the agent’s own outcomes.

A routing decision includes a reasoning budget as well as a model. We encode that budget as an ordered variable, from a terse answer to extended working, and score every model at every setting. When a provider exposes reasoning or verbosity controls, we use them. When it does not, fixed instructions ask for anything from a bare answer to fuller working. R2-Router^[8] takes a related approach, routing over both model and answer-length budget. The same model can land at several different quality-cost points depending on how you call it.

Once model and budget are inputs, routing is no longer a classification problem. For every query and model-budget pair, we regress the outcomes of that call. On RouterArena, those outcomes are the response’s accuracy score and its cost.

Once you write down the training set, it is a tabular regression problem. Each row pairs the query representation with one candidate model’s capability fingerprint and one reasoning budget. The targets are accuracy and cost. TabM^[9] was a natural backbone. It performs strongly on TabArena^[10], the living benchmark for tabular prediction, and its internal ensemble gives us multiple predictors without duplicating the whole network.

For each candidate action, feature-wise linear modulation (FiLM)^[11] turns its model fingerprint and reasoning budget into scales and shifts that modulate the query representation before it enters the ensemble. The members share the main MLP weights, while keeping lightweight adapters and output heads of their own. Each predicts a mean and variance for accuracy and cost. We train those distributions on scored rollouts, including repeats, with a heteroscedastic likelihood plus an auxiliary rank-aware loss term that rewards the correct ordering of candidate actions.

Those predictions remain separate until the model and reasoning budget are chosen. A nightly batch of low-stakes jobs can favour cheaper models. A customer-facing step whose failure is expensive can justify spending more for accuracy. Changing that tradeoff simply reranks the predictions already in hand. It neither retrains the router nor calls a candidate model again.

The leaderboard froze one trade-off

RouterArena allowed us to submit one version of the router at one cost sensitivity. The Sqwish star in Figure 1 is that submitted policy: one way of turning the router’s accuracy and cost predictions into 8,400 choices. The dotted Sqwish sweep comes from holding the fitted predictor fixed and changing the cost sensitivity ourselves. Those other operating points were never leaderboard entries.

Rank the model pool once, then move the slider. The router predicts accuracy and cost for each model and reasoning-budget choice. Moving the slider changes how those predictions are combined, so the ordering updates immediately without retraining the router or running the predictor again.

The embedded demo stops at ranking. In the standalone playground, you can compare real DeepSeek and Gemini responses and audit our ranking and outcome predictions for yourself. The tokens are on us, within the limits set by a finance team that has now read this paragraph.

RouterArena combines two outcomes, accuracy and cost. A production agent may have more, and their relative importance can change by workflow or even by request. Because LoopOne learns each outcome separately, changing the objective does not discard what the system already knows about task success, latency, cost, or user sentiment.

Is the outcome noisy, or is the router guessing?

Once the objective can move at serving time, an average prediction is not enough. An expected accuracy of 80% might describe a model that is reliably near 80%, or one that alternates between excellent answers and complete failures. Those are very different bets for an agent operator.

In our router, every TabM ensemble member predicts a mean and a variance for both accuracy and cost, as shown in Figure 5. Repeated rollouts do two jobs. They show how widely the outcome of the same model-budget choice can move from one call to the next, and they stop one lucky or unlucky run from teaching the router the wrong ordering near a close decision boundary. Statisticians call the first quantity aleatoric uncertainty. More repeats can estimate it better, but cannot make the underlying response variation disappear. For a workflow that can retry, the distribution helps estimate pass@k. For one in which a single bad call breaks the run, it exposes risk that the mean conceals. Risk tolerance can therefore rerank the same cached predictions, just as cost sensitivity does.

The ensemble members are distinct predictors, with their own lightweight adapters and heads. When they disagree about the expected outcomes, the router has too little relevant evidence to support a stable prediction. We use that disagreement as an estimate of epistemic uncertainty, which is uncertainty caused by limited evidence. In LoopOne, it marks cases worth testing with shadow traffic or a small amount of deliberate exploration.

The distinction changed how we acquired data. High predicted response variance called for repeated rollouts so we could measure the distribution. High ensemble disagreement called for new examples near that request. We had too few of both in the places where the model choice was hardest, so we turned to targeted synthetic data.

Twenty thousand tasks were not enough

Our initial training dataset for RouterArena already had a broad base: roughly 20,000 curated tasks from mathematics, code, science, history, and professional work, each scored across the model and reasoning-budget choices available to the router. The resulting table contained hundreds of thousands of outcomes. The policy learned broad model capabilities and then stalled around the family lookup, sometimes below it. Adding more clean, varied questions from neighbouring sources barely moved the result. To beat that baseline, let alone approach the oracle, we needed examples near the decision boundaries of RouterArena itself. Routing is local in this sense: broad data can tell you what a model is generally good at; only distribution-matched data tells you when to choose it over a close alternative.

RouterArena also publishes 809 of its requests as Sub-10, a small subset that was recommended at the time for local testing. It gave us something those broad corpora did not: a sample of the benchmark’s local structure. The no-training rule had not yet been written down, and R2-Router^[12] and other public entries disclosed training or calibration on it. We used those requests as anchors for synthetic generation and as a development validation set. The fitted router trained on synthetic descendants, never the anchors themselves. Sub-10 sits inside RouterArena’s 8,400 requests; we never touched the other 7,591 for training or validation. Repeated checks on that holdout would simply turn it into another training signal.

We first tried the obvious approach: ask a model to write more tasks like the anchors. It kept the topic but made the tasks easier. Rare facts became familiar ones, multi-part constraints disappeared, and distractors became obvious. Nearly every model got them right. The questions passed ordinary quality checks, but model disagreement collapsed and the router had almost nothing to learn from them.

We changed the pipeline accordingly. A candidate had to require a similar kind and level of work as its anchor, and scoring it had to produce roughly the same pattern of model successes and failures. Topic overlap alone told us very little. Even two obscure history questions can be far apart for routing if one answer is common knowledge to the model pool or one set of distractors is much weaker.

We selected anchors where regret was high or the ensemble disagreed. The generation workflows could research the web and run code, but a plausible question was only the first gate. We checked the answer independently, ran exact-match and embedding-based replay checks, and tested whether the task preserved the work and difficulty of its anchor. Only then did we pay to score the candidate across every model and reasoning-budget choice. If the model success and failure pattern had disappeared, the task went through repair and the checks again, or we discarded it.

That process yielded roughly 10,000 new tasks from Sub-10 anchors. Together with roughly 20,000 curated general-purpose tasks, scored across the model-budget actions and repeated where noise mattered, the final training set contained more than a million scored outcomes across tasks, models, and reasoning budgets.

By then we were optimizing useful tasks per dollar per second, where useful meant surviving every gate and still giving the router something to learn. (This must be how TSMC process engineers feel when they are trying to get as many working chips as possible from each silicon wafer. We assume the engineering sophistication is exactly the same.)

In a real agent, useful data is even sparser. You might see an expensive failure five times, not five hundred. This is why LoopOne starts with the agent’s own traces and outcomes. General-purpose data supplies useful priors; only the product’s traffic shows where one action actually beats another. We generate and validate nearby cases so the policy has enough examples to recognise the same failure when it returns in a different form. Synthetic data amplifies the real signal; it does not replace it.

The next RouterArena

By the end of this project, we had a fairly specific idea of what RouterArena v2 should measure. A benchmark is easiest to criticise after it has treated you kindly. RouterArena has, so here is what we would change.

RouterArena turned a fuzzy research question into an open competition that we could enter, inspect, and argue with. Yifan Lu, Rixin Liu, Jiarong Xing, and the rest of the team maintain it in public, largely for free. We are grateful to them for building it and doing the unglamorous work of keeping it running.

One lesson is procedural. Early RouterArena materials recommended Sub-10 for local testing without an explicit training policy, so entrants made different good-faith calls about what was permitted; one full-split submission was later withdrawn^[13]. The README now states the rule plainly^[14]: RouterArena is evaluation-only. That is a much cleaner boundary.

The harder problem is structural. The leaderboard is live; its 8,400 prompts are not. Any fixed, public evaluation eventually becomes familiar through fine-tuning, feature engineering, repeated submissions, or simple exposure. RouterArena is hardly alone. On July 8, OpenAI estimated that roughly 30% of SWE-bench Pro was broken and retracted its earlier recommendation to adopt it^[15], only months after presenting it as the successor to another saturated coding benchmark. Maintaining a hard, fair benchmark is difficult even with vastly more resources.

We would separate two experiments in the next RouterArena. A locked-pool track would give every router the same candidate models and report the fraction of oracle headroom recovered over a simple baseline. If hindsight gains ten utility points over the family lookup and a router finds six of them, it has recovered 60%. That score measures how much routing value the algorithm found and gives nearly tied actions nearly equal credit. Reporting it at several cost sensitivities would keep one arbitrary trade-off from deciding the race.

An open-pool systems track would answer a different question: who can build the best deployable cost-quality policy with the models available to them? Today the two questions are mixed. A newly released model can lift a submission simply by being cheaper and smarter, even if the routing algorithm has not improved. That is a real systems win, but it is not the same experiment.

The next version should also account for router-side compute and stop freezing the decision at the opening prompt. In a real agent, useful evidence may arrive after a tool call, a repository search, or a cheap model’s first failed attempt. The benchmark should follow the trajectory and grade completed outcomes, total spend, and reliability. Applied Compute^[16] and TwinRouterBench^[17] are already testing parts of this design.

RouterArena’s maintainers have already shown why we would trust them with that redesign. When we reported missing reasoning-token costs and zero-usage rows^[18], they confirmed the gaps, patched the evaluator, added validation, and rescored the affected entries within four days^[19]. If RouterArena v2 happens, we hope we get to help build the benchmark that makes our current router obsolete.

Production does not wait for v2. Models change, traffic shifts, and the objective can differ by workflow or by request. The agent’s own traces and outcomes are the only evaluation set that stays current. LoopOne uses those traces to find the next experiment worth running: the one most likely to improve the agent’s next decision.

If your agent leaves traces and you can name an outcome you really want to improve, join the LoopOne early access. We’ll start with the first decision those traces can teach it to make better.

References

1. Xunzhuo Liu and the vLLM Semantic Router team. vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models. arXiv technical report, 2026. See also the v0.3 Themis release note. ↩
2. Hao Li and colleagues. LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing. Findings of ACL, 2026. ↩
3. Seongho Son and colleagues. SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks. Deep Learning for Code Workshop at ICML, 2026. ↩
4. Oscar Skean and colleagues. Layer by Layer: Uncovering Hidden Representations in Language Models. ICML, 2025. ↩
5. Tanay Varshney and colleagues. LLM Router: Rethinking Routing with Prefill Activations. arXiv preprint, 2026. ↩
6. Wes Gurnee and colleagues at Anthropic. Verbalizable Representations Form a Global Workspace in Language Models. Transformer Circuits research paper, July 2026. See also the short research note. ↩
7. Artificial Analysis. Comparison of AI Models across Intelligence, Performance, and Price. Living model-evaluation site, accessed July 2026. ↩
8. Jiaqi Xue, Qian Lou, Jiarong Xing, and Heng Huang. R2-Router: A New Paradigm for LLM Routing with Reasoning. ICML, 2026. ↩
9. Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling. ICLR, 2025. ↩
10. Nick Erickson and colleagues. TabArena: A Living Benchmark for Machine Learning on Tabular Data. NeurIPS Datasets and Benchmarks, 2025. See also the living benchmark. ↩
11. Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual Reasoning with a General Conditioning Layer. AAAI, 2018. ↩
12. Jiaqi Xue and the R2-Router team. R2-Router’s RouterArena submission and model card. Hugging Face, 2026. ↩
13. Yifan Lu and the RouteWorks maintainers. Remove Weave Router from leaderboard per no-training policy. RouterArena pull request 106, May 2026. ↩
14. Yifan Lu and the RouteWorks maintainers. Add no-training policy callout to README. RouterArena pull request 105, May 2026. ↩
15. OpenAI. Separating signal from noise in coding evaluations. OpenAI research publication, July 2026. ↩
16. Applied Compute. Training an Agentic Router for Optimal Cost-Performance on SWE Tasks. Research note, June 2026. ↩
17. Pei Yang and colleagues. TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing. arXiv preprint, 2026. ↩
18. Namitha Kothapalli at Sqwish Labs. Clarify and validate cost accounting for reasoning tokens and empty token_usage rows. RouterArena issue 135, June 2026. ↩
19. Yifan Lu and the RouteWorks maintainers. Fix reasoning-token cost accounting. RouterArena pull request 137, June 2026. ↩
20. Horace He in collaboration with Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Technical post, September 2025. ↩