Abstract:Consumer choices are increasingly mediated by algorithms, which use data on those past choices to infer consumer preferences and then curate future choice sets. Behavioral economics suggests one reason these algorithms so often fail: choices can systematically deviate from preferences. For example, research shows that prejudice can arise not just from preferences and beliefs, but also from the context in which people choose. When people behave automatically, biases creep in; snap decisions are typically more prejudiced than slow, deliberate ones, and can lead to behaviors that users themselves do not consciously want or intend. As a result, algorithms trained on automatic behaviors can misunderstand the prejudice of users: the more automatic the behavior, the greater the error. We empirically test these ideas in a lab experiment, and find that more automatic behavior does indeed seem to lead to more biased algorithms. We then explore the large-scale consequences of this idea by carrying out algorithmic audits of Facebook in its two biggest markets, the US and India, focusing on two algorithms that differ in how users engage with them: News Feed (people interact with friends' posts fairly automatically) and People You May Know (people choose friends fairly deliberately). We find significant out-group bias in the News Feed algorithm (e.g., whites are less likely to be shown Black friends' posts, and Muslims less likely to be shown Hindu friends' posts), but no detectable bias in the PYMK algorithm. Together, these results suggest a need to rethink how large-scale algorithms use data on human behavior, especially in online contexts where so much of the measured behavior might be quite automatic.Institutional subscribers to the NBER working paper series, and residents of developing countries may download this paper without additional charge at www.nber.org.

Competitive Model Selection in Algorithmic Targeting

Dynamic Modeling of Nontargeted and Targeted Advertising Strategies in an Oligopoly

Artificial Intelligence, Algorithmic Competition and Market Structures

Behavior-based algorithmic pricing

Optimizing Scalable Targeted Marketing Policies with Constraints

A Model of Competitive Assortment Planning Algorithm

Comparing Targeting Strategies for Maximizing Social Welfare with Limited Resources

Bias-Variance Games

The Perils of Exploration under Competition: A Computational Modeling Approach

Competitive Machine Learning: Best Theoretical Prediction vs Optimization

Automating Automaticity: How the Context of Human Choice Affects the Extent of Algorithmic Bias

A comparison of methods for model selection when estimating individual treatment effects

Monoculture in Matching Markets

A Decision Theoretic Approach to Targeted Advertising

Social Networks and the Choices People Make

Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning

Dynamically Tie the Right Offer to the Right Customer in Telecommunications Industry

A novel response model and target selection method with applications to marketing

Price Competition Under A Consider-Then-Choose Model With Lexicographic Choice

Time-varying targeted marketing in networks based on competitive opinion dynamics

Adversarial competition and collusion in algorithmic markets