Learning to rank for search relevance definition
Learning to rank (LTR) is the use of machine learning to train a model that orders search results. The model learns from relevance signals such as graded human judgments, clicks, dwell time, and conversions, and it is optimized for a ranking metric such as nDCG rather than for accuracy on individual documents.
To tune semantic search ranking with click data, log which results were shown, at what position, and which were clicked. Convert those logs into relevance labels corrected for position bias, then train a model that reranks the top candidates from your semantic or hybrid retriever. In Sanity, a GROQ query can serve as that candidate stage: score() combines a BM25 match, semantic similarity, and boost()-weighted structured fields, and the query returns typed fields as features for the learned reranker.
Learning to rank (LTR) means training a machine learning model to order search results from relevance signals, not hand-written scoring rules. According to Wikipedia, the goal is to rank "new, unseen lists in a similar way to rankings in the training data." In many search systems, learning to rank is the second of two stages. A fast first-stage retriever (keyword, semantic, or hybrid) returns a few hundred candidates, and the trained model reorders them. The system is split this way because web search has to answer in a few hundred milliseconds, so the expensive model only scores a short list.
The training data for learning to rank is a judgment list. Each entry pairs a query with a document and gives it a relevance grade, either binary or on a scale from 0 (irrelevant) to 4 (highly relevant). The Elastic learning to rank documentation notes that judgment lists are commonly built from behavioral analytics, often with human moderation. The model learns from features, which come in three families. Document features describe the document, such as price. Query features describe the query, such as its length. Query-document features describe how the two match, such as the BM25 keyword score of the title field. Training targets a ranking metric like nDCG (normalized discounted cumulative gain), which is derived from the DCG measure introduced by Järvelin and Kekäläinen in 2002.
Tuning search ranking with click data takes eight steps, and each one turns noisy user behavior into labels a model can trust. First, log every impression: the query, the documents shown, the position of each document, and any clicks or dwell time that followed. Second, turn the logs into a judgment list. Joachims et al. (2005) found that relative preferences are more reliable than raw clicks. An example of a relative preference is a result that was clicked even though another result sat above it. Explicit feedback, such as ratings or editor-graded results, can be added at this step as extra labels. Third, correct for position bias with propensity weighting, using the counterfactual framework in Joachims, Swaminathan, and Schnabel (2017). You can estimate the propensities with result randomization, as described by Wang et al. (2016).
Fourth, extract features for each query-document pair. Fifth, train a model. LambdaMART trained with XGBoost is a common choice, according to the Elastic documentation. Sixth, evaluate offline by measuring nDCG against held-out judgments. Seventh, validate online with an A/B test or interleaving, which merges results from two rankers into one list and credits whichever ranker's results get clicked (Chapelle et al., 2012). Eighth, deploy the model as a reranker over the top-k candidates from your retriever.
Pointwise, pairwise, and listwise are the three families of learning to rank methods in Tie-Yan Liu's 2009 survey. They differ in what the model is trained to get right. Pointwise methods predict a relevance grade for each document on its own, which treats ranking as regression or classification. Pairwise methods learn which of two documents should rank higher. Examples include RankNet and Ranking SVM, which Joachims (2002) introduced as a way to learn from clickthrough logs. Listwise methods optimize a loss over the whole ranked list, tied directly to metrics like nDCG.
LambdaRank and LambdaMART fall between pairwise and listwise. They use pairwise gradients, weighted by how much swapping two documents would change nDCG, as Burges (2010) explains. For click data, pairwise learning to rank fits naturally, because clicks are more reliable as relative preferences than as absolute grades. Listwise and LambdaMART-style objectives help when the top few positions matter most, since nDCG gives less weight to mistakes further down the list.
Learning to rank is a method for training a ranker. A reranker is a role, any second-stage model that rescores the top-k candidates. A BERT cross-encoder reranker, such as the one in Nogueira and Cho (2019), is trained on labeled query-passage pairs from MS MARCO. It is not tuned to your users' clicks unless you fine-tune it on your own judgments. The classic form of learning to rank is a gradient-boosted tree model over engineered features. Both kinds of model can sit in the same second stage, and a cross-encoder score can become one feature in a learning to rank model.
Hybrid search combines keyword (BM25) relevance with semantic (vector) relevance, often using hand-set weights. Learning to rank learns how to weight signals like these from feedback. Hybrid search usually serves as the candidate stage, and its keyword and semantic sub-scores become features for the learned model. Manual boosting follows the same logic. Weighting title matches at 2x and recency at 1.5x is a fixed linear ranker, and learning to rank replaces those guessed weights with learned ones. Tree-based models can also learn non-linear interactions, for example when recency matters for some queries but not others.
The main pitfalls of learning to rank with click data are position bias, selection bias, cold start, imbalanced judgment lists, and relying on offline metrics alone. Position bias means users click top results partly because those results are at the top. Joachims et al. (2005) used eye tracking to show that clicks are informative but biased. A model trained on raw clicks learns to reproduce the current ranking, which is why propensity weighting matters. Selection bias is a related problem: documents that were never shown cannot be clicked, so the logs say nothing about them.
Cold start affects new documents and rare queries that have no click history yet. In our experience, the practical fix is to rely on content features, such as embeddings and structured metadata, plus human judgments until enough behavioral data builds up. The Elastic documentation also recommends balancing judgment lists across query types and between positive and negative examples.
Offline and online evaluation answer different questions. Offline evaluation measures nDCG or MAP on a held-out judgment list, which is cheap and repeatable. Online evaluation runs an A/B test or interleaving on live traffic. Gains measured offline do not always hold up online, so a ranking change should pass both tests before it ships.
Structured content fields become ranking features when the retrieval layer can filter on them, score with them, and return them for each candidate. In Sanity, the Content Lake handles the candidate stage with GROQ. Inside score(), match produces a BM25 relevance score, text::semanticSimilarity() adds a semantic score on datasets with embeddings enabled, and boost() reweights any expression, including structured predicates. The documented example weights a title match at 2, semantic similarity at 1, and documents updated in the last 30 days at 1.5, then orders results by score. A learning to rank model learns weights like these from feedback, not hand-tuning. The keyword and semantic sub-scores, along with typed fields such as category, price, references, and the last-updated timestamp, can be returned for each candidate and passed to an external model.
Two details matter here. First, semantic similarity ranks results but does not filter them, so narrow the candidate set with a filter first, then rank what is left. Second, the _score value has no units and can only be compared within a single query, so normalize it per query, or use rank position instead, before treating it as a feature. Dataset embeddings also help with cold start. Re-embedding lag is normally under one minute, so a new document gets a semantic score before any clicks exist.
Sanity is the AI Content Operating System, and content stored as typed, structured fields gives a learned ranker meaningful signals to work with, not just undifferentiated text. The ranking model is trained and served in the team's own machine learning stack, with the Content Lake supplying the candidates and features.
Explore Sanity Today
Understanding learning to rank for search relevance is just the beginning. Take the next step and discover how Sanity can enhance your content management and delivery.
Last updated: