Abstract:Quantifying variable importance is essential for answering high-stakes questions in fields like genetics, public policy, and medicine. Current methods generally calculate variable importance for a given model trained on a given dataset. However, for a given dataset, there may be many models that explain the target outcome equally well; without accounting for all possible explanations, different researchers may arrive at many conflicting yet equally valid conclusions given the same data. Additionally, even when accounting for all possible explanations for a given dataset, these insights may not generalize because not all good explanations are stable across reasonable data perturbations. We propose a new variable importance framework that quantifies the importance of a variable across the set of all good models and is stable across the data distribution. Our framework is extremely flexible and can be integrated with most existing model classes and global variable importance metrics. We demonstrate through experiments that our framework recovers variable importance rankings for complex simulation setups where other methods fail. Further, we show that our framework accurately estimates the true importance of a variable for the underlying data distribution. We provide theoretical guarantees on the consistency and finite sample error rates for our estimator. Finally, we demonstrate its utility with a real-world case study exploring which genes are important for predicting HIV load in persons with HIV, highlighting an important gene that has not previously been studied in connection with HIV. Code is available at <a class="link-external link-https" href="https://github.com/jdonnelly36/Rashomon_Importance_Distribution" rel="external noopener nofollow">this https URL</a>.

An exact game-theoretic variable importance index for generalized additive models

Statistical Inference for Generalized Additive Models: Simultaneous Confidence Corridors and Variable Selection

Model free variable importance for high dimensional data

The Importance of Variable Importance

Explaining black box decisions by Shapley cohort refinement

Shapley Marginal Surplus for Strong Models

Evaluating the relative importance of predictors in Generalized Additive Models using the gam.hp R package

The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importance

On Bayesian Generalized Additive Models

Proportional Marginal Effects for Global Sensitivity Analysis

Scalable GAM using sparse variational Gaussian processes

Assessing Variable Importance for Best Subset Selection

Bayesian views of generalized additive modelling

Asymptotic theory for extreme value generalized additive models

Shapley variable importance cloud for interpretable machine learning

Generalized Sparse Additive Model with Unknown Link Function

Variable importance in regression models

Generalized additive models with flexible response functions

From global to local MDI variable importances for random forests and when they are Shapley values

Reliable and scalable variable importance estimation via warm-start and early stopping

How Interpretable and Trustworthy are GAMs?