Abstract:Data is central to the development and evaluation of machine learning (ML) models. However, the use of problematic or inappropriate datasets can result in harms when the resulting models are deployed. To encourage responsible AI practice through more deliberate reflection on datasets and transparency around the processes by which they are created, researchers and practitioners have begun to advocate for increased data documentation and have proposed several data documentation frameworks. However, there is little research on whether these data documentation frameworks meet the needs of ML practitioners, who both create and consume datasets. To address this gap, we set out to understand ML practitioners' data documentation perceptions, needs, challenges, and desiderata, with the ultimate goal of deriving design requirements that can inform future data documentation frameworks. We conducted a series of semi-structured interviews with 14 ML practitioners at a single large, international technology company. We had them answer a list of questions taken from datasheets for datasets~\citegebru2018datasheets. Our findings show that current approaches to data documentation are largely ad hoc and myopic in nature. Participants expressed needs for data documentation frameworks to be adaptable to their contexts, integrated into their existing tools and workflows, and automated wherever possible. Despite the fact that data documentation frameworks are often motivated from the perspective of responsible AI, participants did not make the connection between the questions that they were asked to answer and their responsible AI implications. In addition, participants often had difficulties prioritizing the needs of dataset consumers and providing information that someone unfamiliar with their datasets might need to know. Based on these findings, we derive seven design requirements for future data documentation frameworks such as more actionable guidance on how the characteristics of datasets might result in harms and how these harms might be mitigated, more explicit prompts for reflection, automated adaptation to different contexts, and integration into ML practitioners' existing tools and workflows.

Datasheets for Datasets

Datasheet for Subjective and Objective Quality Assessment Datasets

Healthsheet: Development of a Transparency Artifact for Health Datasets

Datasheets for Machine Learning Sensors: Towards Transparency, Auditability, and Responsibility for Intelligent Sensing

Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus

Understanding Machine Learning Practitioners' Data Documentation Perceptions, Needs, Challenges, and Desiderata

Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI

Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on Hugging Face

Data and its (dis)contents: A survey of dataset development and use in machine learning research

Building Better Datasets: Seven Recommendations for Responsible Design from Dataset Creators

A domain-specific language for describing machine learning datasets

The social construction of datasets: On the practices, processes, and challenges of dataset creation for machine learning

Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development

Datasets of Visualization for Machine Learning

Structured dataset documentation: a datasheet for CheXpert

Dataset search: a survey

DSDL: Data Set Description Language for Bridging Modalities and Tasks in AI Data

Talking datasets: Understanding data sensemaking behaviours

Benchmark Data Repositories for Better Benchmarking

AI Competitions and Benchmarks: Dataset Development

Dataset Management Platform for Machine Learning