Experimental Results On Letor

Tie-Yan Liu

DOI: https://doi.org/10.1007/978-3-642-14267-3_11

2011-01-01

Abstract:In this chapter, we take the official evaluation results published at the LETOR website as the source to perform discussions on the performances of different learning-to-rank methods.

What problem does this paper attempt to address?

LETOR: A Benchmark Collection for Research on Learning to Rank for Information Retrieval.

Tao Qin,Tie-Yan Liu,Jun Xu,Hang Li

DOI: https://doi.org/10.1007/s10791-009-9123-y

2010-01-01

Information Retrieval

Abstract:LETOR is a benchmark collection for the research on learning to rank for information retrieval, released by Microsoft Research Asia. In this paper, we describe the details of the LETOR collection and show how it can be used in different kinds of researches. Specifically, we describe how the document corpora and query sets in LETOR are selected, how the documents are sampled, how the learning features and meta information are extracted, and how the datasets are partitioned for comprehensive evaluation. We then compare several state-of-the-art learning to rank algorithms on LETOR, report their ranking performances, and make discussions on the results. After that, we discuss possible new research topics that can be supported by LETOR, in addition to algorithm comparison. We hope that this paper can help people to gain deeper understanding of LETOR, and enable more interesting research projects on learning to rank and related topics.
LETOR: Benchmark Dataset for Research on Learning to Rank for Information Retrieval

Tie-Yan Liu,Jun Xu,Tao Qin,Wenying Xiong,Hang Li

2007-01-01

Abstract:This paper is concerned with learning to rank for information retrieval (IR). Ranking is the central problem for information retrieval, and employing machine learning techniques to learn the ranking function is viewed as a promising approach to IR. Unfortunately, there was no benchmark dataset that could be used in comparison of existing learning algorithms and in evaluation of newly proposed algorithms, which stood in the way of the related research. To deal with the problem, we have constructed a benchmark dataset referred to as LETOR and distributed it to the research communities. Specifically we have derived the LETOR data from the existing data sets widely used in IR, namely, OHSUMED and TREC data. The two collections contain queries, the contents of the retrieved documents, and human judgments on the relevance of the documents with respect to the queries. We have extracted features from the datasets, including both conventional features, such as term frequency, inverse document frequency, BM25, and language models for IR, and features proposed recently at SIGIR, such as HostRank, feature propagation, and topical PageRank. We have then packaged LETOR with the extracted features, queries, and relevance judgments. We have also provided the results of several state-of- the-arts learning to rank algorithms on the data. This paper describes in details about LETOR.
Is learning to rank effective for Web search?

Min Zhang,Da Kuang,Guichun Hua,Yiqun Liu,Shaoping Ma

2009-01-01

Abstract:LETOR, the benchmark collection for learning to rank, helps make comparative study on different approaches in experimental research. Since the collection is constructed mainly based on TREC datasets, queries and documents in LETOR differ from true Web search scenario on some aspects, such as its incomplete link information, limited documents' domain, and lack of user click information. Hence the observations derived by the collection could be different from that in real Web environment. This paper empirically studies the effectiveness of the state-of-art learning to rank algorithms, especially in Web search scenario. Besides LETOR, a Web search collection is constructed based on the search log of a commercial search engine. Five approaches have been studied, including linear regression, RankBoost, ListNet, top k optimization of ListMLE, and SVM-MAP. Comparative study has been made among algorithms and across different datasets. Furthermore, the effects of learning to rank algorithms are compared with that of content-based and link-based ranking features. Essential differences have been observed and analyzed in the paper in terms of the effectiveness and stability of the algorithms and the feature selection. We believe this study will help the Web search community for better knowledge about whether and to what extent learning to rank algorithms are effective in real applications.
Introducing LETOR 4.0 Datasets.

Tao Qin,Tie-Yan Liu

2013-01-01

Abstract: LETOR is a package of benchmark data sets for research on LEarning TO Rank, which contains standard features, relevance judgments, data partitioning, evaluation tools, and several baselines. Version 1.0 was released in April 2007. Version 2.0 was released in Dec. 2007. Version 3.0 was released in Dec. 2008. This version, 4.0, was released in July 2009. Very different from previous versions (V3.0 is an update based on V2.0 and V2.0 is an update based on V1.0), LETOR4.0 is a totally new release. It uses the Gov2 web page collection (~25M pages) and two query sets from Million Query track of TREC 2007 and TREC 2008. We call the two query sets MQ2007 and MQ2008 for short. There are about 1700 queries in MQ2007 with labeled documents and about 800 queries in MQ2008 with labeled documents. If you have any questions or suggestions about the datasets, please kindly email us (letor@microsoft.com). Our goal is to make the dataset reliable and useful for the community.
RankFormer: Listwise Learning-to-Rank Using Listwide Labels

Maarten Buyl,Paul Missault,Pierre-Antoine Sondag

DOI: https://doi.org/10.1145/3580305.3599892

2023-06-09

Abstract:Web applications where users are presented with a limited selection of items have long employed ranking models to put the most relevant results first. Any feedback received from users is typically assumed to reflect a relative judgement on the utility of items, e.g. a user clicking on an item only implies it is better than items not clicked in the same ranked list. Hence, the objectives optimized in Learning-to-Rank (LTR) tend to be pairwise or listwise. Yet, by only viewing feedback as relative, we neglect the user's absolute feedback on the list's overall quality, e.g. when no items in the selection are clicked. We thus reconsider the standard LTR paradigm and argue the benefits of learning from this listwide signal. To this end, we propose the RankFormer as an architecture that, with a Transformer at its core, can jointly optimize a novel listwide assessment objective and a traditional listwise LTR objective. We simulate implicit feedback on public datasets and observe that the RankFormer succeeds in benefitting from listwide signals. Additionally, we conduct experiments in e-commerce on Amazon Search data and find the RankFormer to be superior to all baselines offline. An online experiment shows that knowledge distillation can be used to find immediate practical use for the RankFormer.

Information Retrieval,Machine Learning
Learning to Rank from Relevance Judgments Distributions

Alberto Purpura,Gianmaria Silvello,Gian Antonio Susto

DOI: https://doi.org/10.48550/arXiv.2202.06337

2022-02-13

Abstract:Learning to Rank (LETOR) algorithms are usually trained on annotated corpora where a single relevance label is assigned to each available document-topic pair. Within the Cranfield framework, relevance labels result from merging either multiple expertly curated or crowdsourced human assessments. In this paper, we explore how to train LETOR models with relevance judgments distributions (either real or synthetically generated) assigned to document-topic pairs instead of single-valued relevance labels. We propose five new probabilistic loss functions to deal with the higher expressive power provided by relevance judgments distributions and show how they can be applied both to neural and GBM architectures. Moreover, we show how training a LETOR model on a sampled version of the relevance judgments from certain probability distributions can improve its performance when relying either on traditional or probabilistic loss functions. Finally, we validate our hypothesis on real-world crowdsourced relevance judgments distributions. Overall, we observe that relying on relevance judgments distributions to train different LETOR models can boost their performance and even outperform strong baselines such as LambdaMART on several test collections.

Information Retrieval
A Simple yet Effective Framework for Active Learning to Rank

Qingzhong Wang,Haifang Li,Haoyi Xiong,Wen Wang,Jiang Bian,Yu Lu,Shuaiqiang Wang,Zhicong Cheng,Dejing Dou,Dawei Yin

DOI: https://doi.org/10.1007/s11633-023-1422-z

2023-02-14

Abstract:While China has become the biggest online market in the world with around 1 billion internet users, Baidu runs the world largest Chinese search engine serving more than hundreds of millions of daily active users and responding billions queries per day. To handle the diverse query requests from users at web-scale, Baidu has done tremendous efforts in understanding users' queries, retrieve relevant contents from a pool of trillions of webpages, and rank the most relevant webpages on the top of results. Among these components used in Baidu search, learning to rank (LTR) plays a critical role and we need to timely label an extremely large number of queries together with relevant webpages to train and update the online LTR models. To reduce the costs and time consumption of queries/webpages labeling, we study the problem of Activ Learning to Rank (active LTR) that selects unlabeled queries for annotation and training in this work. Specifically, we first investigate the criterion -- Ranking Entropy (RE) characterizing the entropy of relevant webpages under a query produced by a sequence of online LTR models updated by different checkpoints, using a Query-By-Committee (QBC) method. Then, we explore a new criterion namely Prediction Variances (PV) that measures the variance of prediction results for all relevant webpages under a query. Our empirical studies find that RE may favor low-frequency queries from the pool for labeling while PV prioritizing high-frequency queries more. Finally, we combine these two complementary criteria as the sample selection strategies for active learning. Extensive experiments with comparisons to baseline algorithms show that the proposed approach could train LTR models achieving higher Discounted Cumulative Gain (i.e., the relative improvement {\Delta}DCG4=1.38%) with the same budgeted labeling efforts.

Information Retrieval
AliExpress Learning-To-Rank: Maximizing Online Model Performance without Going Online

Guangda Huzhang,Zhen-Jia Pang,Yongqing Gao,Yawen Liu,Weijie Shen,Wen-Ji Zhou,Qing Da,An-Xiang Zeng,Han Yu,Yang Yu,Zhi-Hua Zhou

DOI: https://doi.org/10.48550/arXiv.2003.11941

2020-12-31

Abstract:Learning-to-rank (LTR) has become a key technology in E-commerce applications. Most existing LTR approaches follow a supervised learning paradigm from offline labeled data collected from the online system. However, it has been noticed that previous LTR models can have a good validation performance over offline validation data but have a poor online performance, and vice versa, which implies a possible large inconsistency between the offline and online evaluation. We investigate and confirm in this paper that such inconsistency exists and can have a significant impact on AliExpress Search. Reasons for the inconsistency include the ignorance of item context during the learning, and the offline data set is insufficient for learning the context. Therefore, this paper proposes an evaluator-generator framework for LTR with item context. The framework consists of an evaluator that generalizes to evaluate recommendations involving the context, and a generator that maximizes the evaluator score by reinforcement learning, and a discriminator that ensures the generalization of the evaluator. Extensive experiments in simulation environments and AliExpress Search online system show that, firstly, the classic data-based metrics on the offline dataset can show significant inconsistency with online performance, and can even be misleading. Secondly, the proposed evaluator score is significantly more consistent with the online performance than common ranking metrics. Finally, as the consequence, our method achieves a significant improvement (\textgreater$2\%$) in terms of Conversion Rate (CR) over the industrial-level fine-tuned model in online A/B tests.

Machine Learning,Artificial Intelligence
An Empirical Perspective on Learning-to-rank.

Fan Li,Wenyu Chen,Zhiyou Yang,Mingsheng Fu,Yun Zhan,Hong Qu

DOI: https://doi.org/10.1145/3594315.3594351

2023-01-01

Abstract:Learning-to-rank has been widely studied and applied in document retrieval. Typically, existing learning-to-rank methods treat ranking as an independent matching process among different queries. Hence, their ranking functions remain unchanged once the learning process is accomplished. However, these methods ignore the fact that previous ranking performance could help fine-tune the ranking function. In a contrast, this paper proposes an empirical perspective-based learning-to-rank model (EPRank) by formulating ranking among different queries as a sequential ranking process where previous ranking feedback could help refine the ranking function for future ranking. Specifically, EPRank constructs a clip feedback mapping where the ranking feedback is mapped to distinct feature embeddings to roughly represent ranking performance for efficient learning. Then, EPRank explicitly incorporates previous queries’ clip feedback embeddings upon earlier ranking performance into a recurrent neural network to iteratively refine the ranking function. Experiments demonstrate the effectiveness of EPRank compared to previous learning-to-rank methods in terms of Normalized Discounted Cumulative Gain (NDCG) and Expected Reciprocal Rank (ERR).
Learning-to-Rank with Nested Feedback

Hitesh Sagtani,Olivier Jeunen,Aleksei Ustimenko

2024-01-09

Abstract:Many platforms on the web present ranked lists of content to users, typically optimized for engagement-, satisfaction- or retention- driven metrics. Advances in the Learning-to-Rank (LTR) research literature have enabled rapid growth in this application area. Several popular interfaces now include nested lists, where users can enter a 2nd-level feed via any given 1st-level item. Naturally, this has implications for evaluation metrics, objective functions, and the ranking policies we wish to learn. We propose a theoretically grounded method to incorporate 2nd-level feedback into any 1st-level ranking model. Online experiments on a large-scale recommendation system confirm our theoretical findings.

Information Retrieval
Online Learning to Rank in a Listwise Approach for Information Retrieval

Fan Ma,Haoyun Yang,Haibing Yin,Xiaofeng Huang,Chenggang Yan,Xiang Meng

DOI: https://doi.org/10.1109/icme.2019.00181

2019-01-01

Abstract:A common approach to learning to rank is to minimize the pair-wise loss. However, established analysis shows that pair-wise loss does not necessarily lead to an optimal list-wise ranking measures, e.g., average precision (AP) or area under precision-recall curve (AUPRC). It becomes more difficult in the online learning setting, where the data arrives sequentially and is scanned only once. This paper proposes an online learning-to-rank algorithm by minimizing the list-wise ranking error, which achieves a vanishing gap between the list-wise loss and the ranking measures. Experiments also testify the effectiveness and robustness of the proposed online List-wise algorithm.
An Attention-Based Interactive Learning-to-Rank Model for Document Retrieval

Fan Zhang,Wenyu Chen,Mingsheng Fu,Fan Li,Hong Qu,Zhang Yi

DOI: https://doi.org/10.1109/TSMC.2021.3129839

2022-01-01

IEEE Transactions on Systems, Man, and Cybernetics: Systems

Abstract:The core issue of learning-to-rank (LTR) for document retrieval lies in finding an optimal ranking policy to meet the search intent of the user. The majority of proposed LTR approaches treat the ranking as a static process, employing a fixed ranking policy to immediately assign scores to documents. By contrast, ranking is not a static but an interactive process where the user continues interacting with the document retrieval system through information exchange such as search intent (e.g., rating or clicking for the retrieved items). We model the interactive ranking process (IRP), and propose an Attention-Based Interactive LTR model (AIRank) to constitute an intent-aware flexible ranking policy to gratify the user's need. To enhance the ranking quality, the inherent relations among documents are procured by the self-attention method to contribute to an enriched user intent representation. Furthermore, we mend the policy gradient learning method to train the AIRank in the IRP. Experiments demonstrate the effectiveness of AIRank compared to the state-of-the-art methods in terms of normalized discounted cumulative gain and expected reciprocal rank.
To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions

Rolf Jagerman,Harrie Oosterhuis,Maarten de Rijke

DOI: https://doi.org/10.1145/3331184.3331269

2019-07-15

Abstract:Learning to Rank (LTR) from user interactions is challenging as user feedback often contains high levels of bias and noise. At the moment, two methodologies for dealing with bias prevail in the field of LTR: counterfactual methods that learn from historical data and model user behavior to deal with biases; and online methods that perform interventions to deal with bias but use no explicit user models. For practitioners the decision between either methodology is very important because of its direct impact on end users. Nevertheless, there has never been a direct comparison between these two approaches to unbiased LTR. In this study we provide the first benchmarking of both counterfactual and online LTR methods under different experimental conditions. Our results show that the choice between the methodologies is consequential and depends on the presence of selection bias, and the degree of position bias and interaction noise. In settings with little bias or noise counterfactual methods can obtain the highest ranking performance; however, in other circumstances their optimization can be detrimental to the user experience. Conversely, online methods are very robust to bias and noise but require control over the displayed rankings. Our findings confirm and contradict existing expectations on the impact of model-based and intervention-based methods in LTR, and allow practitioners to make an informed decision between the two methodologies.

Information Retrieval
PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations

Ruosen Li,Teerth Patel,Xinya Du

2024-07-03

Abstract:Nowadays, the quality of responses generated by different modern large language models (LLMs) is hard to evaluate and compare automatically. Recent studies suggest and predominantly use LLMs for reference-free evaluation of open-ended question answering. More specifically, they use the recognized "strongest" LLM as the evaluator, which conducts pairwise comparisons of candidate models' answers and provides a ranking score. However, this intuitive method has multiple problems, such as bringing in self-enhancement (favoring its own answers) and positional bias. We draw insights and lessons from the educational domain (Cho & MacArthur, 2011; Walsh, 2014) to improve LLM-based evaluations. Specifically, we propose (1) the peer rank (PR) algorithm that takes into account each peer LLM's pairwise preferences of all answer pairs, and outputs a final ranking of models; and (2) peer discussion (PD), where we prompt two LLMs to discuss and try to reach a mutual agreement on the preferences of two answers. We conduct experiments on two benchmark datasets. We find that our approaches achieve higher accuracy and align better with human judgments. Interestingly, PR can induce a relatively accurate self-ranking of models under the anonymous setting, where each model's name is unrevealed. Our work provides space to explore evaluating models that are hard to compare for humans.

Computation and Language,Artificial Intelligence
Future Directions in Learning to Rank.

Olivier Chapelle,Yi Chang,Tie-Yan Liu

2010-01-01

Abstract:The results of the learning to rank challenge showed that the quality of the predictions from the top competitors are very close from each other. This raises a question: is learning to rank a solved problem? On the on hand, it is likely that only small incremental progress can be made in the core and traditional problematics of learning to rank. The challenge was set in this standard learning to rank scenario: optimize a ranking measure on a test set. But on the other hand, there are a lot of related questions and settings in learning to rank that have not been yet fully explored. We review some of them in this paper and hope that researchers interested in learning to rank will try to answer these challenging and exciting research questions.
Lero: applying learning-to-rank in query optimizer

Xingguang Chen,Rong Zhu,Bolin Ding,Sibo Wang,Jingren Zhou

DOI: https://doi.org/10.1007/s00778-024-00850-3

2024-04-26

The VLDB Journal

Abstract:In recent studies, machine learning techniques have been employed to support or enhance cost-based query optimizers in DBMS. Although these approaches have shown superiority in certain benchmarks, they also suffer from certain drawbacks. These include unstable performance, high training costs, and slow model updating, which can be attributed to the inherent challenges of predicting the cost or latency of execution plans using machine learning models. In this paper, we introduce a le arning-to- r ank query o ptimizer, called Lero , which builds on top of the native query optimizer and continuously learns to improve query optimization. The key observation is that the relative order or rank of plans, rather than the exact cost or latency, is sufficient for query optimization. Lero employs a pairwise approach to train a classifier to compare any two plans and tell which one is better. Such a binary classification task is much easier than the regression task to predict the cost or latency, in terms of model efficiency and effectiveness. Rather than building a learned optimizer from scratch, Lero is designed to leverage decades of wisdom of databases and improve the native optimizer. With its non-intrusive design, Lero can be implemented on top of any existing DBMS with minimum integration efforts. We implement Lero and demonstrate its outstanding performance using PostgreSQL and Spark SQL. In our experiments, Lero achieves near-optimal performance on several benchmarks. It reduces the execution time of the native PostgreSQL optimizer by up to and other learned query optimizers by up to on single-machine environments. On distributed environments, our Lero improves the running time of the native Spark SQL optimizer by up to . Meanwhile, Lero continuously learns and automatically adapts to query workloads and changes in data.

computer science, information systems, hardware & architecture
On Application of Learning to Rank for E-Commerce Search

Shubhra Kanti Karmaker Santu,Parikshit Sondhi,ChengXiang Zhai

DOI: https://doi.org/10.1145/3077136.3080838

2019-03-02

Abstract:E-Commerce (E-Com) search is an emerging important new application of information retrieval. Learning to Rank (LETOR) is a general effective strategy for optimizing search engines, and is thus also a key technology for E-Com search. While the use of LETOR for web search has been well studied, its use for E-Com search has not yet been well explored. In this paper, we discuss the practical challenges in applying learning to rank methods to E-Com search, including the challenges in feature representation, obtaining reliable relevance judgments, and optimally exploiting multiple user feedback signals such as click rates, add-to-cart ratios, order rates, and revenue. We study these new challenges using experiments on industry data sets and report several interesting findings that can provide guidance on how to optimally apply LETOR to E-Com search: First, popularity-based features defined solely on product items are very useful and LETOR methods were able to effectively optimize their combination with relevance-based features. Second, query attribute sparsity raises challenges for LETOR, and selecting features to reduce/avoid sparsity is beneficial. Third, while crowdsourcing is often useful for obtaining relevance judgments for Web search, it does not work as well for E-Com search due to difficulty in eliciting sufficiently fine grained relevance judgments. Finally, among the multiple feedback signals, the order rate is found to be the most robust training objective, followed by click rate, while add-to-cart ratio seems least robust, suggesting that an effective practical strategy may be to initially use click rates for training and gradually shift to using order rates as they become available.

Information Retrieval,Machine Learning
Learning to Rank in Generative Retrieval

Yongqi Li,Nan Yang,Liang Wang,Furu Wei,Wenjie Li

DOI: https://doi.org/10.1609/aaai.v38i8.28717

2024-03-24

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:Generative retrieval stands out as a promising new paradigm in text retrieval that aims to generate identifier strings of relevant passages as the retrieval target. This generative paradigm taps into powerful generative language models, distinct from traditional sparse or dense retrieval methods. However, only learning to generate is insufficient for generative retrieval. Generative retrieval learns to generate identifiers of relevant passages as an intermediate goal and then converts predicted identifiers into the final passage rank list. The disconnect between the learning objective of autoregressive models and the desired passage ranking target leads to a learning gap. To bridge this gap, we propose a learning-to-rank framework for generative retrieval, dubbed LTRGR. LTRGR enables generative retrieval to learn to rank passages directly, optimizing the autoregressive model toward the final passage ranking target via a rank loss. This framework only requires an additional learning-to-rank training phase to enhance current generative retrieval systems and does not add any burden to the inference stage. We conducted experiments on three public benchmarks, and the results demonstrate that LTRGR achieves state-of-the-art performance among generative retrieval methods. The code and checkpoints are released at https://github.com/liyongqi67/LTRGR.

English Else
Learning to Rank Broad and Narrow Queries in E-Commerce

Siddhartha Devapujula,Sagar Arora,Sumit Borar

DOI: https://doi.org/10.48550/arXiv.1907.01549

2019-07-15

Abstract:Search is a prominent channel for discovering products on an e-commerce platform. Ranking products retrieved from search becomes crucial to address customer's need and optimize for business metrics. While learning to Rank (LETOR) models have been extensively studied and have demonstrated efficacy in the context of web search; it is a relatively new research area to be explored in the e-commerce. In this paper, we present a framework for building LETOR model for an e-commerce platform. We analyze user queries and propose a mechanism to segment queries between broad and narrow based on user's intent. We discuss different types of features - query, product and query-product and discuss challenges in using them. We show that sparsity in product features can be tackled through a denoising auto-encoder while skip-gram based word embeddings help solve the query-product sparsity issues. We also present various target metrics that can be employed for evaluating search results and compare their robustness. Further, we build and compare performances of both pointwise and pairwise LETOR models on fashion category data set. We also build and compare distinct models for broad and narrow queries, analyze feature importance across these and show that these specialized models perform better than a combined model in the fashion world.

Information Retrieval,Computation and Language,Machine Learning
Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application

Yujing Hu,Qing Da,Anxiang Zeng,Yang Yu,Yinghui Xu

DOI: https://doi.org/10.48550/arXiv.1803.00710

2018-05-23

Abstract:In e-commerce platforms such as Amazon and TaoBao, ranking items in a search session is a typical multi-step decision-making problem. Learning to rank (LTR) methods have been widely applied to ranking problems. However, such methods often consider different ranking steps in a session to be independent, which conversely may be highly correlated to each other. For better utilizing the correlation between different ranking steps, in this paper, we propose to use reinforcement learning (RL) to learn an optimal ranking policy which maximizes the expected accumulative rewards in a search session. Firstly, we formally define the concept of search session Markov decision process (SSMDP) to formulate the multi-step ranking problem. Secondly, we analyze the property of SSMDP and theoretically prove the necessity of maximizing accumulative rewards. Lastly, we propose a novel policy gradient algorithm for learning an optimal ranking policy, which is able to deal with the problem of high reward variance and unbalanced reward distribution of an SSMDP. Experiments are conducted in simulation and TaoBao search engine. The results demonstrate that our algorithm performs much better than online LTR methods, with more than 40% and 30% growth of total transaction amount in the simulation and the real application, respectively.

Machine Learning

Experimental Results On Letor

LETOR: A Benchmark Collection for Research on Learning to Rank for Information Retrieval.

LETOR: Benchmark Dataset for Research on Learning to Rank for Information Retrieval

Is learning to rank effective for Web search?

Introducing LETOR 4.0 Datasets.

RankFormer: Listwise Learning-to-Rank Using Listwide Labels

Learning to Rank from Relevance Judgments Distributions

A Simple yet Effective Framework for Active Learning to Rank

AliExpress Learning-To-Rank: Maximizing Online Model Performance without Going Online

An Empirical Perspective on Learning-to-rank.

Learning-to-Rank with Nested Feedback

Online Learning to Rank in a Listwise Approach for Information Retrieval

An Attention-Based Interactive Learning-to-Rank Model for Document Retrieval

To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions

PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations

Future Directions in Learning to Rank.

Lero: applying learning-to-rank in query optimizer

On Application of Learning to Rank for E-Commerce Search

Learning to Rank in Generative Retrieval

Learning to Rank Broad and Narrow Queries in E-Commerce

Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application