Abstract:Assessing the quality and impact of individual data points is critical for improving model performance and mitigating undesirable biases within the training dataset. Several data valuation algorithms have been proposed to quantify data quality, however, there lacks a systemic and standardized benchmarking system for data valuation. In this paper, we introduce OpenDataVal, an easy-to-use and unified benchmark framework that empowers researchers and practitioners to apply and compare various data valuation algorithms. OpenDataVal provides an integrated environment that includes (i) a diverse collection of image, natural language, and tabular datasets, (ii) implementations of eleven different state-of-the-art data valuation algorithms, and (iii) a prediction model API that can import any models in scikit-learn. Furthermore, we propose four downstream machine learning tasks for evaluating the quality of data values. We perform benchmarking analysis using OpenDataVal, quantifying and comparing the efficacy of state-of-the-art data valuation approaches. We find that no single algorithm performs uniformly best across all tasks, and an appropriate algorithm should be employed for a user's downstream task. OpenDataVal is publicly available at <a class="link-external link-https" href="https://opendataval.github.io" rel="external noopener nofollow">this https URL</a> with comprehensive documentation. Furthermore, we provide a leaderboard where researchers can evaluate the effectiveness of their own data valuation algorithms.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to evaluate the quality and impact of individual data points in order to improve model performance and mitigate adverse biases in the training dataset. Specifically, the paper proposes a unified benchmark framework named OpenDataVal, aiming to provide researchers and practitioners with an easy - to - use and standardized environment for applying and comparing various data valuation algorithms. OpenDataVal offers diverse datasets, multiple state - of - the - art data valuation algorithm implementations, and a prediction model API. At the same time, it also proposes four downstream machine - learning tasks to evaluate the quality of data values. ### Background and Problem of the Paper In the real world, data usually shows heterogeneity and noise. These data come from a wide range of sources and are vulnerable to measurement and annotation errors. When such data is incorporated into model development, it may introduce adverse biases, which will hinder model performance. Managing and solving these problems are crucial for ensuring the reliability and accuracy of insights drawn from real data. Therefore, it becomes increasingly important to understand and evaluate the intrinsic properties of data, such as data quality, data bias, and their impact on the model training process. Motivated by these, data valuation emerges as a principled evaluation framework, aiming to quantify the impact of individual data points on model prediction or model performance. ### Main Contributions 1. **Unified Benchmark Framework**: OpenDataVal provides a unified benchmark framework that enables researchers and practitioners to easily apply and compare different data valuation algorithms. 2. **Diverse Datasets**: It includes image, natural language, and tabular datasets, which are from open platforms such as OpenML, scikit - learn, and PyTorch. 3. **Multiple Data Valuation Algorithms**: 11 different state - of - the - art data valuation algorithms are implemented. 4. **Prediction Model API**: It supports importing any scikit - learn and PyTorch - based neural network models. 5. **Downstream Tasks**: Four downstream machine - learning tasks are provided to evaluate the quality of data values, including noisy label detection, noisy feature detection, point - removal experiment, and point - addition experiment. 6. **Open and Transparent**: OpenDataVal is open - source and provides a leaderboard, and users can participate in the noisy data detection competition. ### Experimental Results Through benchmark analysis, the paper finds that no data valuation algorithm performs best in all tasks, and different algorithms have their own advantages in different tasks. For example: - **Data - OOB** performs well in noisy label detection but poorly in the point - removal experiment. - **LAVA** stands out in noisy feature detection. - **AME, DataShapley, BetaShapley, and KNNShapley** perform well in the point - removal experiment. ### Conclusion OpenDataVal provides a unified and easy - to - use benchmark framework for data valuation research, promoting the transparency and reproducibility of data valuation methods. Users can choose the appropriate algorithm according to their downstream tasks to improve model performance and data quality.

OpenDataVal: a Unified Benchmark for Data Valuation

EcoVal: An Efficient Data Valuation Framework for Machine Learning

Neural Dynamic Data Valuation

LAVA: Data Valuation without Pre-Specified Learning Algorithms

Is Data Valuation Learnable and Interpretable?

Data Valuation by Leveraging Global and Local Statistical Information

Data Distribution Valuation

Fundamentals of Task-Agnostic Data Valuation

Data Valuation from Data-Driven Optimization

DeRDaVa: Deletion-Robust Data Valuation for Machine Learning

When Crowdsourcing Meets Data Markets: A Fair Data Value Metric for Data Trading

Data Valuation with Gradient Similarity

DataPerf: Benchmarks for Data-Centric AI Development

Benchmark Data Repositories for Better Benchmarking

Personalization of Dataset Retrieval Results using a Metadata-based Data Valuation Method

GMValuator: Similarity-based Data Valuation for Generative Models

SAVA: Scalable Learning-Agnostic Data Valuation

WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models

2D-OOB: Attributing Data Contribution Through Joint Valuation Framework

Private Data Valuation and Fair Payment in Data Marketplaces

OpenPerf: A Benchmarking Framework for the Sustainable Development of the Open-Source Ecosystem