Abstract:Ethics statements have been proposed as a mechanism to increase transparency and promote reflection on the societal impacts of published research. In 2020, the machine learning (ML) conference NeurIPS broke new ground by requiring that all papers include a broader impact statement. This requirement was removed in 2021, in favour of a checklist approach. The 2020 statements therefore provide a unique opportunity to learn from the broader impact experiment: to investigate the benefits and challenges of this and similar governance mechanisms, as well as providing an insight into how ML researchers think about the societal impacts of their own work. Such learning is needed as NeurIPS and other venues continue to question and adapt their policies. To enable this, we have created a dataset containing the impact statements from all NeurIPS 2020 papers, along with additional information such as affiliation type, location and subject area, and a simple visualisation tool for exploration. We also provide an initial quantitative analysis of the dataset, covering representation, engagement, common themes, and willingness to discuss potential harms alongside benefits. We investigate how these vary by geography, affiliation type and subject area. Drawing on these findings, we discuss the potential benefits and negative outcomes of ethics statement requirements, and their possible causes and associated challenges. These lead us to several lessons to be learnt from the 2020 requirement: (i) the importance of creating the right incentives, (ii) the need for clear expectations and guidance, and (iii) the importance of transparency and constructive deliberation. We encourage other researchers to use our dataset to provide additional analysis, to further our understanding of how researchers responded to this requirement, and to investigate the benefits and challenges of this and related mechanisms.

Ethical Considerations and Statistical Analysis of Industry Involvement in Machine Learning Research

AI Ethics Statements -- Analysis and lessons learnt from NeurIPS Broader Impact Statements

The Values Encoded in Machine Learning Research

Ethical Machine Learning in Health Care

The ethical ambiguity of AI data enrichment: Measuring gaps in research ethics norms and practices

Machine learning and AI research for Patient Benefit: 20 Critical Questions on Transparency, Replicability, Ethics and Effectiveness

Ethics and Governance of Artificial Intelligence: Evidence from a Survey of Machine Learning Researchers

The Elephant in the Room: Analyzing the Presence of Big Tech in Natural Language Processing Research

Improving fairness in machine learning systems: What do industry practitioners need?

State of the Art in Fair ML: From Moral Philosophy and Legislation to Fair Classifiers

Ethics and Governance of Artificial Intelligence: A Survey of Machine Learning Researchers (extended Abstract)

Evaluating the Economic Implications of Using Machine Learning in Clinical Psychiatry

Evaluation Gaps in Machine Learning Practice

Ethical Considerations in AI Addressing Bias and Fairness in Machine Learning Models

Can everyday AI be ethical. Fairness of Machine Learning Algorithms

Can Machine Learning be Moral?

A Conceptual Framework for Ethical Evaluation of Machine Learning Systems

Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness

Some Ethical Issues in the Review Process of Machine Learning Conferences

A Comprehensive Study of Groundbreaking Machine Learning Research: Analyzing highly cited and impactful publications across six decades

The Narrow Depth and Breadth of Corporate Responsible AI Research