Abstract:Interleaving is an online evaluation approach for information retrieval systems that compares the effectiveness of ranking functions in interpreting the users' implicit feedback. Previous work such as Hofmann et al (2011) has evaluated the most promising interleaved methods at the time, on uniform distributions of queries. In the real world, ordinarily, there is an unbalanced distribution of repeated queries that follows a long-tailed users' search demand curve. The more a query is executed, by different users (or in different sessions), the higher the probability of collecting implicit feedback (interactions/clicks) on the related search results. This paper first aims to replicate the Team Draft Interleaving accuracy evaluation on uniform query distributions and then focuses on assessing how this method generalizes to long-tailed real-world scenarios. The reproducibility work raised interesting considerations on how the winning ranking function for each query should impact the overall winner for the entire evaluation. Based on what was observed, we propose that not all the queries should contribute to the final decision in equal proportion. As a result of these insights, we designed two variations of the $\Delta_{AB}$ score winner estimator that assign to each query a credit based on statistical hypothesis testing. To replicate, reproduce and extend the original work, we have developed from scratch a system that simulates a search engine and users' interactions from datasets from the industry. Our experiments confirm our intuition and show that our methods are promising in terms of accuracy, sensitivity, and robustness to noise.

Joint Use of Multiple Learned Statistics for Improving Online Source Selection

Mining, Using and Maintaining Source Statistics for Adaptive Data Integration

Automatic Accuracy Assessment Via Hashing in Multiple-Source Environment

Assessing Accuracy with Locality-Sensitive Hashing in Multiple Source Environment.

Mining and using coverage and overlap statistics for data integration

Effectively Mining and Using Coverage and Overlap Statistics for Data Integration

Mining Coverage Statistics for Websource Selection in a Mediator

Mining Source Coverage Statistics for Data Integration

Social network mashup: Ontology-based social network integration for statistic learning

Joint Optimization of Cost and Coverage of Query Plans in Data Integration

Data source selection for information integration in big data era

Joinable Search over Multi-source Spatial Datasets: Overlap, Coverage, and Efficiency

Learning to Seek: Multi-Agent Online Source Seeking Against Non-Stochastic Disturbances

Quality of Information-Based Source Assessment and Selection.

Selecting Sources for Query Approximation with Bounded Resources.

Efficient Quality-Driven Source Selection from Massive Data Sources

Data Source Selection for Approximate Query

Stat-weight: Improving the Estimator of Interleaved Methods Outcomes with Statistical Hypothesis Testing

Effective Keyword-Based Selection of Relational Databases.

Optimizing Source Selection in Social Sensing in the Presence of Influence Graphs

Learning More Powerful Test Statistics for Click-Based Retrieval Evaluation