Abstract:The hyperparameters of recommender systems for top-n predictions are typically optimized to enhance the predictive performance of algorithms. Thereby, the optimization algorithm, e.g., grid search or random search, searches for the best hyperparameter configuration according to an optimization-target metric, like nDCG or Precision. In contrast, the optimized algorithm, internally optimizes a different loss function during training, like squared error or cross-entropy. To tackle this discrepancy, recent work focused on generating loss functions better suited for recommender systems. Yet, when evaluating an algorithm using a top-n metric during optimization, another discrepancy between the optimization-target metric and the training loss has so far been ignored. During optimization, the top-n items are selected for computing a top-n metric; ignoring that the top-n items are selected from the recommendations of a model trained with an entirely different loss function. Item recommendations suitable for optimization-target metrics could be outside the top-n recommended items; hiddenly impacting the optimization performance. Therefore, we were motivated to analyze whether the top-n items are optimal for optimization-target top-n metrics. In pursuit of an answer, we exhaustively evaluate the predictive performance of 250 selection strategies besides selecting the top-n. We extensively evaluate each selection strategy over twelve implicit feedback and eight explicit feedback data sets with eleven recommender systems algorithms. Our results show that there exist selection strategies other than top-n that increase predictive performance for various algorithms and recommendation domains. However, the performance of the top ~43% of selection strategies is not significantly different. We discuss the impact of our findings on optimization and re-ranking in recommender systems and feasible solutions.

Personalization in E-Grocery: Top-N versus Top-k Rankings

Page-level Optimization of e-Commerce Item Recommendations

Performance of recommender algorithms on top-n recommendation tasks

Top-Personalized-K Recommendation

Personalized Review Ranking for Improving Shopper's Decision Making: A Term Frequency based Approach

Serendipitous Personalized Ranking For Top-N Recommendation

Revealing the Hidden Impact of Top-N Metrics on Optimization in Recommender Systems

Addressing Purchase-Impression Gap through a Sequential Re-ranker

Building a Scalable, Effective, and Steerable Search and Ranking Platform

Optimizing Gross Merchandise Volume via DNN-MAB Dynamic Ranking Paradigm

Beyond Movie Recommendations: Solving the Continuous Cold Start Problem in E-commerceRecommendations

Learning Personalized Page Content Ranking Using Customer Representation

UNDR: User-Needs-Driven Ranking of Products in E-Commerce

Niche Product Retrieval in Top-N Recommendation

Order Effects in Online Product Recommendation: A Scenario-based Analysis.

Product Ranking for Revenue Maximization with Multiple Purchases

Personalized Product Assortment with Real-time 3D Perception and Bayesian Payoff Estimation

Personalized Transformer-based Ranking for e-Commerce at Yandex

End-to-End Neural Ranking for eCommerce Product Search: an application of task models and textual embeddings

Rethinking the Role of Pre-ranking in Large-scale E-Commerce Searching System

Incorporating Consumer Browse Data: Extended Item-Based Top-K Recommendation Algorithms