Abstract:Deep learning-based recommender systems (DRSs) are increasingly and widely deployed in the industry, which brings significant convenience to people’s daily life in different ways. However, recommender systems are also shown to suffer from multiple issues, e.g., the echo chamber and the Matthew effect , of which the notation of “fairness” plays a core role. For instance, the system may be regarded as unfair to 1) a specific user, if the user gets worse recommendations than other users, or 2) an item (to recommend), if the item is much less likely to be exposed to the users than other items. While many fairness notations and corresponding fairness testing approaches have been developed for traditional deep classification models, they are essentially hardly applicable to DRSs. One major challenge is that there still lacks a systematic understanding and mapping between the existing fairness notations and the diverse testing requirements for deep recommender systems, not to mention further testing or debugging activities. To address the gap, we propose FairRec, a unified framework that supports fairness testing of DRSs from multiple customized perspectives, e.g., model utility, item diversity, item popularity, etc. We also propose a novel, efficient search-based testing approach to tackle the new challenge, i.e., double-ended discrete particle swarm optimization (DPSO) algorithm, to effectively search for hidden fairness issues in the form of certain disadvantaged groups from a vast number of candidate groups. Given the testing report, by adopting a simple re-ranking mitigation strategy on these identified disadvantaged groups, we show that the fairness of DRSs can be significantly improved. We conducted extensive experiments on multiple industry-level DRSs adopted by leading companies. The results confirm that FairRec is effective and efficient in identifying the deeply hidden fairness issues, e.g., achieving ∼95% testing accuracy with ∼half to 1/8 time.

From Literature to Practice: Exploring Fairness Testing Tools for the Software Industry Adoption

Fairness testing: testing software for discrimination

FairRec: Fairness Testing for Deep Recommender Systems

Fairness Testing: A Comprehensive Survey and Analysis of Trends

Software Fairness Debt

Navigating Fairness: Practitioners' Understanding, Challenges, and Strategies in AI/ML Development

This Thing Called Fairness: Disciplinary Confusion Realizing a Value in Technology

Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support

Preliminary Insights on Industry Practices for Addressing Fairness Debt

How fair are we? From conceptualization to automated assessment of fairness definitions

FairLay-ML: Intuitive Debugging of Fairness in Data-Driven Social-Critical Software

From Expectation to Habit: Why Do Software Practitioners Adopt Fairness Toolkits?

Towards Responsible AI: A Design Space Exploration of Human-Centered Artificial Intelligence User Interfaces to Investigate Fairness

FairCompass: Operationalising Fairness in Machine Learning

Improving fairness in machine learning systems: What do industry practitioners need?

Fair by design: A sociotechnical approach to justifying the fairness of AI-enabled systems across the lifecycle

Understanding Fairness in Software Engineering: Insights from Stack Exchange

AI Fairness in Practice

The Right Tool for the Job: Open-Source Auditing Tools in Machine Learning

Assessing Perceived Fairness from Machine Learning Developer's Perspective

AI Fairness: from Principles to Practice