Abstract:Computer vision technologies have been applied to an increasingly wide range of applications from autonomous car navigation, to medical image analysis, to precision agriculture. Despite many of these exciting innovations, recent studies reveal a number of risks in using existing computer vision systems, suggesting results of such systems may be unfair or untrustworthy. For example, major commercial facial analysis tools were shown to have substantial accuracy disparities for people of different gender or with different skin colors (Buolamwini and Gebru 2018). Visual semantic role labeling models were found to exhibit societal biases and stereotypes (Zhao et al. 2017) such as frequently associating certain activity labels with specific gender (e.g., associate “cooking” with woman). Even worse, seemly accurate image classifiers may in fact made the predictions by picking up spurious correlations between objects and irrelevant background information rather than identifying meaningful features of the objects (Ribeiro, Singh, and Guestrin 2016). Many of the risks embedded in modern computer vision systems can be attributed to the use of a training dataset that is biased. Indeed, the computer vision community has long recognized that many visual datasets present varying degrees of build-in bias due to factors such as photographic style of photographers and selection from dataset curators (Torralba, Efros, and others 2011). Using these biased datasets to train machine learning models for addressing different computer vision tasks naturally leads to the phenomenon of “bias in, bias out” and results in undesirable performance. Thus, to mitigate the fairness, accountability, and transparency concerns in computer vision, a crucial step is to start the entire pipeline with high-quality visual datasets that, at least, are authentic representations of the visual world. In other words, being able to detect potential biases hidden in the datasets prior to model development is a key step in guarding against unfair or untrustworthy outcomes in computer vision. While a few techniques have been developed to automatically detect dataset biases (Tramer et al. 2017), the nonstructured nature of visual data makes bias discovery in image datasets particularly challenging. This is because no human-comprehensive attributes can be directly leveraged

BIAS: Transparent reporting of biomedical image analysis challenges

How can we do better ? Pitfalls in biomedical challenge design and how to address them

Is the winner really the best? A critical analysis of common research practice in biomedical image analysis competitions.

Discovering Biases in Image Datasets with the Crowd

Toward fairness in artificial intelligence for medical image analysis: identification and mitigation of potential biases in the roadmap from data collection to model deployment

How can we learn (more) from challenges? A statistical approach to driving future algorithm development

Beyond the Leaderboard: Insight and Deployment Challenges to Address Research Problems

Challenge Results Are Not Reproducible

The Measurement and Mitigation of Algorithmic Bias and Unfairness in Healthcare AI Models Developed for the CMS AI Health Outcomes Challenge

Beyond rankings: Learning (more) from algorithm validation

Tackling algorithmic bias and promoting transparency in health datasets: the STANDING Together consensus recommendations

Measuring Risk of Bias in Biomedical Reports: The RoBBR Benchmark

Why Rankings of Biomedical Image Analysis Competitions Should Be Interpreted with Care

Towards objective and systematic evaluation of bias in artificial intelligence for medical imaging

Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models

The crucial role of bioimage analysts in scientific research and publication

Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects

Reducing Biases towards Minoritized Populations in Medical Curricular Content via Artificial Intelligence for Fairer Health Outcomes

Understanding metric-related pitfalls in image analysis validation

A Framework for Challenge Design: Insight and Deployment Challenges to Address Medical Image Analysis Problems