LLM routing is the practice of directing each incoming query to whichever language model in a fleet is best suited to handle it, rather than sending every request to a single fixed endpoint. A routing layer evaluates signals such as query complexity, required domain knowledge, expected output length, cost per token, and latency constraints, then dispatches the request accordingly. The goal is to preserve quality on hard tasks while avoiding the unnecessary expense and latency of using the most powerful model for every simple request.

Think of it like a hospital triage desk. A trained nurse quickly assesses each patient and sends minor ailments to a general-practitioner clinic while directing serious cases to a specialist ward. The specialist's time is preserved for cases that genuinely need it.

A typical router might classify a query as "simple" (factual lookup, short answer) or "complex" (multi-step reasoning, code generation) and act on that classification:

def route(query: str) -> str:
    if classifier.predict(query) == "simple":
        return call_model("gpt-4o-mini", query)   # cheap, fast
    else:
        return call_model("claude-opus-4", query)  # capable, costly

Open-source routers like RouteLLM train a small binary classifier on human preference data to make this split, reporting that up to 40% of queries can be served by the cheaper model with negligible quality loss. More sophisticated systems use cascades: the cheap model answers first, and a confidence score determines whether to escalate to a larger model.

LLM routing depends on accurate Inference cost modelling and is often paired with Prompt Engineering to reframe queries before forwarding them. It relates to Mixture of Experts architecture at the model level, where routing happens inside a single model rather than between separate deployments. LLM capability benchmarks inform the routing policy, and Zero-Shot versus Chain of Thought prompting strategies may differ between routed tiers.

You will encounter LLM routing when building production AI systems where budget and response-time SLAs must both be satisfied across a high volume of diverse queries.