Effects of AI Feedback on Learning, the Skill Gap, and Intellectual Diversity

Christoph Riedl,Eric Bogert

2024-09-27

Abstract:Can human decision-makers learn from AI feedback? Using data on 52,000 decision-makers from a large online chess platform, we investigate how their AI use affects three interrelated long-term outcomes: Learning, skill gap, and diversity of decision strategies. First, we show that individuals are far more likely to seek AI feedback in situations in which they experienced success rather than failure. This AI feedback seeking strategy turns out to be detrimental to learning: Feedback on successes decreases future performance, while feedback on failures increases it. Second, higher-skilled decision-makers seek AI feedback more often and are far more likely to seek AI feedback after a failure, and benefit more from AI feedback than lower-skilled individuals. As a result, access to AI feedback increases, rather than decreases, the skill gap between high- and low-skilled individuals. Finally, we leverage 42 major platform updates as natural experiments to show that access to AI feedback causes a decrease in intellectual diversity of the population as individuals tend to specialize in the same areas. Together, those results indicate that learning from AI feedback is not automatic and using AI correctly seems to be a skill itself. Furthermore, despite its individual-level benefits, access to AI feedback can have significant population-level downsides including loss of intellectual diversity and an increasing skill gap.

General Economics,Artificial Intelligence,Human-Computer Interaction

What problem does this paper attempt to address?

The paper attempts to address the following issues:

1. **When human decision-makers seek AI feedback**: Investigating under what circumstances (success or failure) human decision-makers are most likely to seek AI feedback, and whether this tendency varies with different skill levels. Specifically, the authors explore the behavior patterns of high-skilled and low-skilled decision-makers in seeking AI feedback in situations of success and failure.

2. **The impact of AI feedback on learning**: Studying whether AI feedback aids in the long-term learning of decision-makers, particularly if there is a difference in the learning speed from AI feedback between high-skilled and low-skilled decision-makers. This involves comparing the effects of AI feedback in situations of success and failure, and whether such feedback can help bridge the existing skill gap.

3. **Changes in skill gaps due to AI feedback**: Exploring whether the use of AI feedback increases or decreases the skill gap between high-skilled and low-skilled decision-makers. The authors specifically focus on whether AI feedback benefits high-skilled decision-makers more, thereby exacerbating the skill gap.

4. **The long-term impact of AI feedback on intellectual diversity**: Investigating whether AI feedback leads to specialization in specific fields, thereby affecting intellectual diversity at the group level. Specifically, the authors explore whether AI feedback causes individuals to over-specialize in the same field, reducing overall strategic diversity and negatively impacting the long-term problem-solving capabilities of organizations and groups.

Through these issues, the paper aims to comprehensively understand the role of AI feedback in the decision-making process, including its impact on individual learning, skill gaps, and social intellectual diversity.