Abstract:E-commerce search systems such as Taobao Search, the largest e-commerce searching system in China, aim at providing users with the most preferred items (e.g., products). Due to the massive data and limited time for response, a typical industrial ranking system consists of three or more modules, including matching, pre-ranking, and ranking. The pre-ranking is widely considered a mini-ranking module, as it needs to rank hundreds of times more items than the ranking under limited latency. Existing researches focus on building a lighter model that imitates the ranking model. As such, the metric of a pre-ranking model follows the ranking model using Area Under ROC (AUC) for offline evaluation. However, such a metric is inconsistent with online A/B tests in practice, so engineers have to perform costly online tests to reach a convincing conclusion. In our work, we rethink the role of the pre-ranking. We argue that the primary goal of the pre-ranking stage is to return an optimal unordered set rather than an ordered list of items because it is the ranking that determines the final exposures. Since AUC measures the quality of an ordered item list, it is not suitable for evaluating the quality of the output unordered set. This paper proposes a new evaluation metric called All-Scenario Hitrate (ASH) for pre-ranking. ASH is proven effective in the offline evaluation and consistent with online A/B tests based on numerous experiments in Taobao Search. We also introduce an all-scenario-based multi-objective learning framework (ASMOL), which improves the ASH significantly. Surprisingly, the new pre-ranking model can outperforms the ranking model when outputting thousands of items. The phenomenon validates that the pre-ranking stage should not imitate the ranking blindly. With the improvements in ASH consistently translating to online improvement, it makes a 1.2% GMV improvement on Taobao Search.

Rank Aggregation Algorithms Based on Voting Model for Metasearch

Metasearch Via Voting

An End-to-end Model of Predicting Diverse Ranking on Heterogeneous Feeds.

Rank Aggregation Using Scoring Rules

Supervised rank aggregation

Heuristic Search for Rank Aggregation with Application to Label Ranking

QoRank: A Query-Dependent Ranking Model Using LSE-based Weighted Multiple Hyperplanes Aggregation for Information Retrieval

A New Probabilistic Model for Rank Aggregation.

Approximate and Dynamic Rank Aggregation

Listwise Approach For Rank Aggregation In Crowdsourcing

Diversity in Kemeny Rank Aggregation: A Parameterized Approach

SOCIAL CHOICE FOR DATA FUSION

Partition-Mallows Model and Its Inference for Rank Aggregation

The Methods for Ranking Research Projects Based on Common-weights Ranked Voting Model

Application of Rank Aggregation to Campus Network Search Engine

Directly optimizing evaluation measures in learning to rank.

Rethinking the Role of Pre-ranking in Large-scale E-Commerce Searching System

Approximate Rank Aggregation

Surprisingly Popular Voting for Concentric Rank-Order Models

Ranking model selection and fusion for effective microblog search

Monotone Retargeting for Unsupervised Rank Aggregation with Object Features