Perspectives on Machine Learning from Psychology's Reproducibility Crisis

Samuel J. Bell,Onno P. Kampman

DOI: https://doi.org/10.48550/arXiv.2104.08878

2021-04-23

Abstract:In the early 2010s, a crisis of reproducibility rocked the field of psychology. Following a period of reflection, the field has responded with radical reform of its scientific practices. More recently, similar questions about the reproducibility of machine learning research have also come to the fore. In this short paper, we present select ideas from psychology's reformation, translating them into relevance for a machine learning audience.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

This paper attempts to address the issue of reproducibility in the field of machine learning. Specifically, the authors draw lessons from the reproducibility crisis experienced by the field of psychology in the early 2010s, exploring how psychology responded to this crisis by reforming its scientific research practices, and translating these lessons into insights for the machine learning community.

### Main Issues:
1. **Reproducibility Issue**: Can the research results in the field of machine learning be independently verified?
2. **Methodological Issue**: How can experimental design and data analysis methods be improved to enhance the reproducibility and reliability of research?
3. **Cultural Issue**: How can the incentive mechanisms of research culture be changed to promote more reproducible research?

### Specific Measures:
1. **A Priori Hypotheses**: Avoid "HARKing" (Hypothesizing After the Results are Known), which means proposing hypotheses after seeing the results. Researchers should clearly state their hypotheses and predictions before the experiment.
2. **Pre-registration**: Publicly register research hypotheses, experimental design, and analysis plans before conducting experiments to prevent selective reporting and "p-hacking."
3. **Multiverse Analysis**: Evaluate the robustness of research results by considering different data analysis choices.
4. **Publication and Registered Reports**: Adopt a registered report system to reduce publication bias and encourage the publication of negative results.
5. **Incentive Mechanisms**: Adjust the incentive structure for researchers by rewarding reproducible research (e.g., awarding reproducibility badges).
6. **Communication and Collaboration**: Avoid oversimplifying and distorting research results, and encourage more collaboration and replication studies.

### Goal:
By drawing on the experiences of the field of psychology, the aim is to promote reforms in methodology, culture, and theory within the machine learning community to enhance the reproducibility and reliability of research, thereby fostering more robust and credible scientific research.