Abstract:Quantifying the importance of each training point to a learning task is a fundamental problem in machine learning and the estimated importance scores have been leveraged to guide a range of data workflows such as data summarization and domain adaption. One simple idea is to use the leave-one-out error of each training point to indicate its importance. Recent work has also proposed to use the Shapley value, as it defines a unique value distribution scheme that satisfies a set of appealing properties. However, calculating Shapley values is often expensive, which limits its applicability in real-world applications at scale. Multiple heuristics to improve the scalability of calculating Shapley values have been proposed recently, with the potential risk of compromising their utility in real-world applications. \textit{How well do existing data quantification methods perform on existing workflows? How do these methods compare with each other, empirically and theoretically? Must we sacrifice scalability for the utility in these workflows when using these methods?} In this paper, we conduct a novel theoretical analysis comparing the utility of different importance quantification methods, and report extensive experimental studies on existing and proposed workflows such as noisy label detection, watermark removal, data summarization, data acquisition, and domain adaptation. We show that Shapley value approximation based on a $K$NN surrogate over pre-trained feature embeddings obtains comparable utility with existing algorithms while achieving significant scalability improvement, often by orders of magnitude. Our theoretical analysis also justifies its advantage over the leave-one-out error. The code is available at \url{<a class="link-external link-https" href="https://github.com/AI-secure/Shapley-Study" rel="external noopener nofollow">this https URL</a>}.

On the Inflation of KNN-Shapley Value

Absolute Shapley Value

Dynamic Shapley Value Computation.

Towards Data Valuation via Asymmetric Data Shapley

CS-Shapley: Class-wise Shapley Values for Data Valuation in Classification

Evaluate the Contribution of Multiple Participants in Federated Learning

K-Fold Cross-Valuation for Machine Learning Using Shapley Value

Data valuation: The partial ordinal Shapley value for machine learning

CHG Shapley: Efficient Data Valuation and Selection towards Trustworthy Machine Learning

Data Valuation by Leveraging Global and Local Statistical Information

P-Shapley: Shapley Values on Probabilistic Classifiers

Scalability vs. Utility: Do We Have to Sacrifice One for the Other in Data Importance Quantification?

Efficient Data Shapley for Weighted Nearest Neighbor Algorithms

Optimizing Data Shapley Interaction Calculation from O(2^n) to O(t n^2) for KNN models

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

Uncertainty Quantification of Data Shapley via Statistical Inference

Provably Accurate Shapley Value Estimation via Leverage Score Sampling

Data valuation for medical imaging using Shapley value and application to a large-scale chest X-ray dataset

DU-Shapley: A Shapley Value Proxy for Efficient Dataset Valuation

Variance reduced shapley value estimation for trustworthy data valuation