Abstract:Quantifying the importance of each training point to a learning task is a fundamental problem in machine learning and the estimated importance scores have been leveraged to guide a range of data workflows such as data summarization and domain adaption. One simple idea is to use the leave-one-out error of each training point to indicate its importance. Recent work has also proposed to use the Shapley value, as it defines a unique value distribution scheme that satisfies a set of appealing properties. However, calculating Shapley values is often expensive, which limits its applicability in real-world applications at scale. Multiple heuristics to improve the scalability of calculating Shapley values have been proposed recently, with the potential risk of compromising their utility in real-world applications. \textit{How well do existing data quantification methods perform on existing workflows? How do these methods compare with each other, empirically and theoretically? Must we sacrifice scalability for the utility in these workflows when using these methods?} In this paper, we conduct a novel theoretical analysis comparing the utility of different importance quantification methods, and report extensive experimental studies on existing and proposed workflows such as noisy label detection, watermark removal, data summarization, data acquisition, and domain adaptation. We show that Shapley value approximation based on a $K$NN surrogate over pre-trained feature embeddings obtains comparable utility with existing algorithms while achieving significant scalability improvement, often by orders of magnitude. Our theoretical analysis also justifies its advantage over the leave-one-out error. The code is available at \url{<a class="link-external link-https" href="https://github.com/AI-secure/Shapley-Study" rel="external noopener nofollow">this https URL</a>}.

The Importance of Parameters in Database Queries

The Shapley Value in Database Management

Absolute Shapley Value

Scalability vs. Utility: Do We Have to Sacrifice One for the Other in Data Importance Quantification?

A $k$-additive Choquet integral-based approach to approximate the SHAP values for local interpretability in machine learning

Quantifying Feature Importance of Games and Strategies via Shapley Values

RankSHAP: Shapley Value Based Feature Attributions for Learning to Rank

Shapley Marginal Surplus for Strong Models

A Comparative Study of Methods for Estimating Conditional Shapley Values and When to Use Them

CHG Shapley: Efficient Data Valuation and Selection towards Trustworthy Machine Learning

Expected Shapley-Like Scores of Boolean Functions: Complexity and Applications to Probabilistic Databases

Explaining black box decisions by Shapley cohort refinement

P-Shapley: Shapley Values on Probabilistic Classifiers

Approximating the Shapley Value without Marginal Contributions

A Shapley value index on the importance of variables in DEA models

Improving the Sampling Strategy in KernelSHAP

From SHAP Scores to Feature Importance Scores

The Counterfactual-Shapley Value: Attributing Change in System Metrics

Nonparametric Feature Impact and Importance

ShapG: new feature importance method based on the Shapley value

Parameter Screening Using Impact Factors and Surrogate-Based ANOVA Techniques