Abstract:Deep learning-based recommender systems (DRSs) are increasingly and widely deployed in the industry, which brings significant convenience to people’s daily life in different ways. However, recommender systems are also shown to suffer from multiple issues, e.g., the echo chamber and the Matthew effect , of which the notation of “fairness” plays a core role. For instance, the system may be regarded as unfair to 1) a specific user, if the user gets worse recommendations than other users, or 2) an item (to recommend), if the item is much less likely to be exposed to the users than other items. While many fairness notations and corresponding fairness testing approaches have been developed for traditional deep classification models, they are essentially hardly applicable to DRSs. One major challenge is that there still lacks a systematic understanding and mapping between the existing fairness notations and the diverse testing requirements for deep recommender systems, not to mention further testing or debugging activities. To address the gap, we propose FairRec, a unified framework that supports fairness testing of DRSs from multiple customized perspectives, e.g., model utility, item diversity, item popularity, etc. We also propose a novel, efficient search-based testing approach to tackle the new challenge, i.e., double-ended discrete particle swarm optimization (DPSO) algorithm, to effectively search for hidden fairness issues in the form of certain disadvantaged groups from a vast number of candidate groups. Given the testing report, by adopting a simple re-ranking mitigation strategy on these identified disadvantaged groups, we show that the fairness of DRSs can be significantly improved. We conducted extensive experiments on multiple industry-level DRSs adopted by leading companies. The results confirm that FairRec is effective and efficient in identifying the deeply hidden fairness issues, e.g., achieving ∼95% testing accuracy with ∼half to 1/8 time.

RBoard: A Unified Platform for Reproducible and Reusable Recommender System Benchmarks

FairRec: Fairness Testing for Deep Recommender Systems

RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms

Towards a More User-Friendly and Easy-to-Use Benchmark Library for Recommender Systems.

Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison

BARS: Towards Open Benchmarking for Recommender Systems

From Variability to Stability: Advancing RecSys Benchmarking Practices

DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation

RecBole 2.0: Towards a More Up-to-Date Recommendation Library

Revisiting Reciprocal Recommender Systems: Metrics, Formulation, and Method

BETA-Rec: Build, Evaluate and Tune Automated Recommender Systems.

RecStudio: Towards a Highly-Modularized Recommender System.

Revisiting BPR: A Replicability Study of a Common Recommender System Baseline

Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking

Tenrec: A Large-scale Multipurpose Benchmark Dataset for Recommender Systems

A Revisiting Study of Appropriate Offline Evaluation for Top- N Recommendation Algorithms

RBRA: A Simple and Efficient Rating-Based Recommender Algorithm to Cope with Sparsity in Recommender Systems

RecAD: Towards A Unified Library for Recommender Attack and Defense

Collaborative Topic Regression for Online Recommender Systems: an Online and Bayesian Approach

A Reproducible Analysis of Sequential Recommender Systems

A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems