Query routing definition
Query routing is the step in an AI system that decides where a request should go before it is answered: which data source, index, or tool should serve it, and which model should handle it. The decision is made by a separate, usually cheaper component that runs ahead of the main answer.
Query routing in LLM applications is the decision a system makes before it answers: send this request to a structured query against a database, to a search over a prose knowledge base, or to a live API, and have it answered by a small cheap model or a frontier one. In Sanity, that first choice is resolved at the endpoint: an MCP serving GROQ mode queries the dataset at request time, while Knowledge Base mode serves a pre-built index, so an agent inherits the route from configured sources.
A query router decides two things, sometimes both in the same pipeline: which source or tool should serve the request, and which model should handle it.
Source routing picks the destination. A question like "size L latex gloves, under $200 a pallet" is a filter over records, so it belongs on a structured query endpoint. A question like "is this industrial latex food-safe?" may have its answer spread across a specification, a compliance memo, and a support article, so it belongs on a knowledge-base search. In retrieval frameworks this is the router query engine pattern, where a router component sits in front of several query engines and selects one or more per request. LlamaIndex documents it directly, including routing between a vector index, a summary index, and a SQL backend.
Model routing, usually called LLM routing, picks the answering model. Microsoft describes its model router as "a trained language model that intelligently routes your prompts in real time to the most suitable large language model," analyzing each prompt on "complexity, reasoning, task type, and other attributes" (Microsoft Foundry docs). AWS frames its Intelligent Prompt Routing as "a single serverless endpoint to efficiently route requests between different foundation models within the same model family."
The signals a router reads are consistent across both cases: intent or task type, difficulty, how fresh the answer needs to be, the shape of the answer space, and the cost and latency budget for this request. Microsoft exposes the budget part as a dial with three modes, Balanced, Cost, and Quality, where Quality "picks the highest quality rated model for the prompt, ignoring the cost."
Query routers are built in one of five ways, in rough order of cost per decision.
Rules. Keyword, regex, path, or metadata conditions. Cheapest and the easiest to read six months later, and brittle the moment someone phrases a request differently.
Embeddings and similarity. Each route is described by a handful of example utterances, embedded once, and an incoming query is matched against them by vector similarity. The open-source Semantic Router library is the canonical implementation and describes itself as a decision layer that avoids an LLM generation call entirely.
Trained classifiers. A model learned from data about which destination produced better answers. RouteLLM is the standard reference, learning routers from preference data to choose between a strong and a weak model. Later work adds explicit cost terms and many-model settings, including CARROT and Microsoft Research's BEST-Route.
Decision models. A model that returns a typed value over a fixed answer space instead of prose. Picking one destination from a list is a classification problem, not a writing problem, which is why this class fits routing well. Note that a format guarantee is not a correctness guarantee: a router built this way always returns a valid route, and the route can still be wrong.
The main model itself. Hand the LLM the tool menu and let it choose. Simplest to build, and the most expensive per decision, because every routing choice costs a full generation call.
The economics are what pick between these. A router runs on every single request, so its own latency and cost are charged against the savings it produces. That is the whole reason the cheaper families exist.
Query routing is about fit, and the things it gets confused with are about authority, capacity, and sequence.
Tool gating asks whether a call is allowed, and answers allow, deny, or ask a human. Query routing asks which of several permitted destinations will answer best. They touch at catalog filtering, where a large tool catalog is narrowed to the handful plausibly relevant to the request. Narrowing the menu is a gating act with a routing effect. The clean line: gating is about authority and risk, routing is about fit, cost, and latency.
Tool calling is the model emitting a structured request to invoke a named function. Query routing is the decision about which destination is in play at all, and it can happen before the model runs, in a component far cheaper than a generation call.
Load balancing spreads traffic across interchangeable backends for capacity and availability. Query routing picks between options that are not interchangeable, based on what the request is. Routers often add failover on top, which is resilience bolted to the router rather than routing itself.
Cascading runs the cheap model first and escalates when a check fails. Query routing decides once, up front, so it needs an accurate difficulty estimate; a cascade needs none but sometimes pays twice. The two are treated in a unified framework at ICML 2025.
Query rewriting changes the words of a request and decomposition splits it into several. Both sit alongside routing in the pre-retrieval step, and a decomposed question can route each sub-question somewhere different. Reranking reorders results after retrieval; routing chose which retriever ran.
Query routers fail silently, which is the single most important thing to know about operating one. A misrouted request does not throw an exception. It returns a worse answer, so routing problems arrive as diffuse complaints about quality rather than as alerts. DigitalOcean puts it plainly in a vendor-neutral write-up: multi-model routing adds latency and new failure surfaces, and "misrouting manifests as worse answers".
The research literature names several specific pathologies. Routing collapse is when a router degenerates toward always picking the most capable and most expensive model as the budget rises, erasing the savings it was built for (arXiv 2602.03478). Misguided objectives describes how standard cost-accuracy training biases routers toward escalating moderately hard and shorter queries (arXiv 2610.02762). And routers can be systematically fragile by category, making consistently poor decisions on particular kinds of query, with a safety consequence: work presented at EACL 2026 found jailbreak attempts being sent to weaker models.
The design response is to treat the routing decision as a component with its own evaluation rather than as plumbing. Log every decision with the router version, the features it saw, the selected route, its confidence, any fallback that fired, and the eventual outcome, then score those decisions against what the better destination would have returned. Practical checklists for this exist at Polyaxon and MLflow, and Microsoft documents per-request routing metadata and fallbacks as a monitoring surface.
Structured content makes query routing easier by giving the router a bounded input and a bounded answer space, which are the two things that decide whether a routing decision is cheap and reliable or expensive and vague.
A router works best when it is reading typed values rather than inferring a situation from prose. Typed fields, references to taxonomy documents, and document revisions are facts a router can read directly: this document is in review, it belongs to this product category, it was last changed by this person. Handing a router a paragraph of text that describes those same facts is strictly harder work for strictly worse precision. As the AI Content Operating System, Sanity keeps content in that typed form in the Content Lake, and GROQ projections return only the fields named, so the state passed to a routing decision can be the one block it concerns rather than a whole rendered page.
The answer space matters as much as the input. A router is really a choice over a fixed list of destinations, which is why the emerging class of decision models, which return probabilities over a fixed set of options rather than generated text, suits routing, tagging, and classification. The first commercially available one, TypeSafe AI's Jev, was released on 15 September 2026 and takes state plus typed questions and returns probabilities over a defined answer space with no prose. A Choice question over a list of destinations is, structurally, a router.
Sanity resolves the source-routing question at configuration time rather than per request. Sanity Context has two retrieval modes: GROQ mode queries your dataset at request time, and Knowledge Base mode (currently in beta, enabled by an organization admin) serves an index built from your material ahead of time. The mode determines which tools Context MCP serves, and it is derived from the endpoint's configured sources rather than set directly, so an agent inherits a route instead of gambling on one per query. For the middle ground, where a request is part filter and part meaning, dataset embeddings let a single GROQ query combine a structured filter with semantic ranking, which is often a better answer than routing at all.
Explore Sanity Today
Understanding query routing is just the beginning. Take the next step and discover how Sanity can enhance your content management and delivery.
Last updated: