Abstract:Data is central to the development and evaluation of machine learning (ML) models. However, the use of problematic or inappropriate datasets can result in harms when the resulting models are deployed. To encourage responsible AI practice through more deliberate reflection on datasets and transparency around the processes by which they are created, researchers and practitioners have begun to advocate for increased data documentation and have proposed several data documentation frameworks. However, there is little research on whether these data documentation frameworks meet the needs of ML practitioners, who both create and consume datasets. To address this gap, we set out to understand ML practitioners' data documentation perceptions, needs, challenges, and desiderata, with the ultimate goal of deriving design requirements that can inform future data documentation frameworks. We conducted a series of semi-structured interviews with 14 ML practitioners at a single large, international technology company. We had them answer a list of questions taken from datasheets for datasets~\citegebru2018datasheets. Our findings show that current approaches to data documentation are largely ad hoc and myopic in nature. Participants expressed needs for data documentation frameworks to be adaptable to their contexts, integrated into their existing tools and workflows, and automated wherever possible. Despite the fact that data documentation frameworks are often motivated from the perspective of responsible AI, participants did not make the connection between the questions that they were asked to answer and their responsible AI implications. In addition, participants often had difficulties prioritizing the needs of dataset consumers and providing information that someone unfamiliar with their datasets might need to know. Based on these findings, we derive seven design requirements for future data documentation frameworks such as more actionable guidance on how the characteristics of datasets might result in harms and how these harms might be mitigated, more explicit prompts for reflection, automated adaptation to different contexts, and integration into ML practitioners' existing tools and workflows.

Brazilian Data Scientists: Revealing their Challenges and Practices on Machine Learning Model Development

On the Interaction between Software Engineers and Data Scientists when building Machine Learning-Enabled Systems

Towards Effective Collaboration between Software Engineers and Data Scientists developing Machine Learning-Enabled Systems

Industrial Practices of Requirements Engineering for ML-Enabled Systems in Brazil

Characterizing Data Scientists in the Real World

Data Lifecycle Challenges in Production Machine Learning

Beyond the technical challenges for deploying Machine Learning solutions in a software company

Data Science, Machine Learning and Big Data in Digital Journalism: A survey of state-of-the-art, challenges and opportunities

Understanding the Challenges of Data Management in the AI Application Development

Understanding Machine Learning Practitioners' Data Documentation Perceptions, Needs, Challenges, and Desiderata

Data Science Methodologies: Current Challenges and Future Approaches

A Data Quality-Driven View of MLOps

Experimentation, deployment and monitoring Machine Learning models: Approaches for applying MLOps

Towards Productizing AI/ML Models: An Industry Perspective from Data Scientists

Improving Data Quality through Deep Learning and Statistical Models

How Far are we from Data Mining Democratisation? A Systematic Review

Scaling Data Science: Challenges in Enterprise Adoption

The development and deployment of machine learning models

Data-Driven Aerospace Engineering: Reframing the Industry with Machine Learning

Navigating the challenges in creating complex data systems: a development philosophy

Software engineering for artificial intelligence and machine learning software: A systematic literature review