Abstract:Artificial intelligence (AI) systems built on incomplete or biased data will often exhibit problematic outcomes. Current methods of data analysis, particularly before model development, are costly and not standardized. The Dataset Nutrition Label (the Label) is a diagnostic framework that lowers the barrier to standardized data analysis by providing a distilled yet comprehensive overview of dataset "ingredients" before AI model development. Building a Label that can be applied across domains and data types requires that the framework itself be flexible and adaptable; as such, the Label is comprised of diverse qualitative and quantitative modules generated through multiple statistical and probabilistic modelling backends, but displayed in a standardized format. To demonstrate and advance this concept, we generated and published an open source prototype with seven sample modules on the ProPublica Dollars for Docs dataset. The benefits of the Label are manyfold. For data specialists, the Label will drive more robust data analysis practices, provide an efficient way to select the best dataset for their purposes, and increase the overall quality of AI models as a result of more robust training datasets and the ability to check for issues at the time of model development. For those building and publishing datasets, the Label creates an expectation of explanation, which will drive better data collection practices. We also explore the limitations of the Label, including the challenges of generalizing across diverse datasets, and the risk of using "ground truth" data as a comparison dataset. We discuss ways to move forward given the limitations identified. Lastly, we lay out future directions for the Dataset Nutrition Label project, including research and public policy agendas to further advance consideration of the concept.

Data Labeling: An Empirical Investigation into Industrial Challenges and Mitigation Strategies

Understanding the Process of Data Labeling in Cybersecurity

LabelSens: Enabling Real-time Sensor Data Labelling at the point of Collection on Edge Computing

Harnessing label semantics to extract higher performance under noisy label for Company to Industry matching

Automotive Perception Software Development: An Empirical Investigation into Data, Annotation, and Ecosystem Challenges

An Empirical Study of Automated Mislabel Detection in Real World Vision Datasets

Investigating Labeler Bias in Face Annotation for Machine Learning

LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems

Mitigating the impact of mislabeled data on deep predictive models: an empirical study of learning with noise approaches in software engineering tasks

Label Smarter, Not Harder: CleverLabel for Faster Annotation of Ambiguous Image Classification with Higher Quality

Hands-On Tutorial: Labeling with LLM and Human-in-the-Loop

Accurate Label Refinement from Multiannotator of Remote Sensing Data

Automatic Data Labeling for Software Vulnerability Prediction Models: How Far Are We?

No Need to Sacrifice Data Quality for Quantity: Crowd-Informed Machine Annotation for Cost-Effective Understanding of Visual Data

Fixing Mislabeling by Human Annotators Leveraging Conflict Resolution and Prior Knowledge

Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets

The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards

Data Collection and Labeling Techniques for Machine Learning

Active label cleaning for improved dataset quality under resource constraints

Towards a better labeling process for network security datasets

Towards Reducing Data Acquisition and Labeling for Defect Detection using Simulated Data