Hugging Face says model routing isn’t that simple.

On July 14, the official Hugging Face blog published a long article by the IBM Research team titled “Model Routing Is Simple. Until It Isn’t.” The post broke down the much-hyped “model routing” trend that’s been circulating in the agent community over the past year. The conclusion hits hard: routing looks like a money-saving miracle, but in practice, it’s a multi-objective game that balances cost, latency, accuracy, and operational complexity.

If you’re currently building a multi-agent system, or struggling to decide “should I use GPT-5, Claude Sonnet, or DeepSeek” for your product’s LLM integration, this article is worth studying closely.

The Temptation of Routing: An Idea That Sounds Too Cheap to Be True

Let’s start with why everyone is talking about routing.

In the past six months, the performance gap between frontier models has widened, and so has the price gap. Flagship models (like GPT-5, Claude Opus 4.1, or Gemini 2.5 Pro) can cost 20–50× more per call than smaller models, yet for tasks like “translate this sentence into English,” output quality is nearly identical. That naturally leads to this idea: add a classifier in front of requests—send simple problems to small models, complex ones to big models—so you could theoretically save 60–80% on costs while barely losing performance.

On slides, that logic is bulletproof. The IBM team admits that in academic papers, routing looks great—RouteLLM, Hybrid LLM, and FrugalGPT all show benchmark data proving that you can get “95% of the quality for half the cost.”

But after running it in real production for several months, they found things were nothing like that.

Pitfall #1: You Don’t Really Know What “Hard” Means

The core of routing is teaching the classifier to decide “is this query hard?” It sounds like a binary classification or simple regression problem—but what does “hard” actually mean?

IBM’s blog gives a textbook example: the user asks, “What is the capital of France?”—clearly easy. But if they ask, “What controversial decision did the mayor of France’s capital make in 2015?”, the question seems only slightly longer but actually requires a combination of factual recall, temporal reasoning, and political context understanding. Perceived complexity and actual reasoning complexity often diverge.

What’s worse, in multi-turn conversations “difficulty” is dynamic. The first turn may be easy, so the router sends it to a small model; but if the next follow-up question evolves into a deep reasoning task, and you still stick with the small model, all previous dialogue context becomes wasted. If you switch models mid-conversation, the KV cache becomes invalid, and latency and cost spike.

Many teams train a dedicated difficulty classifier, feeding it labeled data to learn “which inputs need a stronger model.” IBM points out a chicken-and-egg problem here: to label difficulty, you must already run multiple models on each query and compare results. In short, to train a router that saves money, you first have to spend a lot to generate training data.

Pitfall #2: Routing Itself Takes Time

This is a detail many papers conveniently ignore.

Routing decisions themselves are inference calls. If you use a small classifier model (say, a fine-tuned BERT or DeBERTa), each decision might take 20–80 ms. If you prioritize accuracy and use a larger LLM (asking GPT‑4o mini to decide where to send a prompt), each decision may take 300 ms or more.

For chat scenarios, users are sensitive to first-token latency. Adding a 300 ms routing pre-step inserts an immediate bottleneck into your request chain. Especially in streaming responses, users can’t see that routing phase—they just feel “this time the model is slower.”

IBM’s data shows that in one enterprise setup, when routing decisions exceeded 15% of total response time, user satisfaction (NPS) dropped significantly—even though output quality didn’t. In other words, the savings from routing can be offset by latency penalties.

Pitfall #3: Models Aren’t Static

This is perhaps the most insightful observation in the whole post.

A trained classifier assumes the back-end model boundaries are fixed. But in reality, OpenAI, Anthropic, and Google silently update their models every couple of months—the same API endpoint and model name may now behave differently.

For example, your classifier six months ago decided that GPT‑4o struggled with math, so all math problems went to o1. But now GPT‑5 mini can already handle most medium-difficulty math tasks at one-fifth the cost. Your classifier doesn’t know this and keeps sending easy math problems to the expensive reasoning model.

It can also go the other way—some teams have found that certain closed-source models degrade after updates (“stealth nerfs”). If your routing logic is hardcoded, you’ll miss this drift entirely.

IBM recommends adding a continuous evaluation mechanism—run a benchmark suite each week covering multiple task types and dynamically adjust routing weights. In other words, routing isn’t a “train once and deploy” feature—it’s a continuously maintained system.

Pitfall #4: Failure Domains Multiply

With a single-model setup, you only worry about one provider going down. With routing, you now depend on 3–5 providers; if any one has issues, the whole routing chain gets disrupted.

IBM describes a real scenario: their production system integrated OpenAI, Anthropic, and self-hosted open-source models. Once, Anthropic’s API suffered regional latency spikes (not downtime—just P99 latency jumping from 2s to 15s). The router kept sending “long-context reasoning” tasks to Claude, and tail latency for the entire app collapsed.

The fix involved adding health probes, fallback chains, and circuit breakers at the routing layer—but once you add all that, the router evolves from “a simple classifier” into “a full service mesh.” Engineering complexity jumps by an order of magnitude.

The Three Schools of Routing

IBM also mapped out the current mainstream routing strategies, roughly divided into three camps:

• Rule-based routing – Hardcode rules using prompt length, keywords, or task type. Pros: zero latency, explainable. Cons: maintenance hell, rule explosion.
• Classifier-based routing – Train a small model to make decisions. Most common today; RouteLLM is the archetype. Works best for limited task domains.
• LLM-based routing – Use an LLM to decide which LLM to call. Most flexible but also the most expensive; suited for true agentic systems where the router is part of the agent.

The author’s take: most companies don’t need complex routing—rule-based routing plus a single model is enough. Only if your traffic volume is huge (millions of daily calls) and savings clearly exceed system maintenance costs should you use classifier-based routing. Below that scale, a single mid-tier model (like GPT‑5 mini or Claude Haiku 4.5) is actually more cost-effective.

An Overlooked Option: Routing via Aggregation Layer

IBM didn’t mention this, but it’s worth adding: if you don’t want to build routing infrastructure yourself, consider using API aggregation platforms to reduce integration burden. Platforms like OpenAI Hub have already standardized GPT, Claude, Gemini, and DeepSeek under OpenAI-compatible APIs—a single key lets you switch providers. Your routing layer only needs to decide what to fill in the model field, without worrying about each provider’s auth, rate limits, or error codes.

This isn’t a direct routing solution, but at least it frees you from the “integrate five SDKs” nightmare, letting you focus on routing logic itself.

Agent Orchestration: Routing Is the Easy Part

The blog ends by extending the topic to multi-agent systems. The IBM team notes frankly: if plain model routing is already this complex, agent orchestration is its “complexity squared.”

Because in agent settings, routing targets not a single call but an entire task trajectory. You must consider:

• Which subtasks should a strong model plan and a weaker one execute?
• When should an agent decide to escalate to a stronger model?
• How to amortize routing decision overhead across multiple calls?
• If an agent fails, retry the same model or upgrade?

There are no standard answers yet. IBM mentions experimenting with a mechanism called “self-aware routing,” allowing agents to assess in real time whether their current model is adequate—and if reasoning chains grow complex, proactively request an upgrade. It’s akin to how operating systems dynamically adjust process priority.

A Bit of Sober Reflection

The biggest takeaway from this blog is: “routing” as a term has been oversimplified by marketing. Vendors selling LLM gateway products love to boast how much money routing saves, but teams that have actually implemented it know that every dollar saved hides monitoring, evaluation, and maintenance overhead.

For AI application teams in 2026, the author’s advice is:

1. Don’t rush into routing. Run with a mid-tier model first and understand your traffic patterns.
2. Prioritize observability. You need per-call insight into cost, latency, and quality before optimizing.
3. Rule-based routing should always be the first choice. If if-else can solve it, don’t bring in ML.
4. Never underestimate model drift. Any routing policy must come with continuous evaluation.

The value of this Hugging Face post is not in offering a new algorithm but in pulling the community out of the illusion that “routing is a silver bullet.” At least for the foreseeable future, model routing remains more of an engineering craft than a pure research problem.

References

• Model Routing Is Simple. Until It Isn’t. – Hugging Face Blog – Original post by IBM Research on Hugging Face, a detailed look at the challenges of model routing in production
• Top 20 Hugging Face Papers in December: Agents, Multimodality, and Code Intelligence – Zhihu – A roundup of recent Hugging Face agent system research, useful background for routing work