Abstract:Quantifying variable importance is essential for answering high-stakes questions in fields like genetics, public policy, and medicine. Current methods generally calculate variable importance for a given model trained on a given dataset. However, for a given dataset, there may be many models that explain the target outcome equally well; without accounting for all possible explanations, different researchers may arrive at many conflicting yet equally valid conclusions given the same data. Additionally, even when accounting for all possible explanations for a given dataset, these insights may not generalize because not all good explanations are stable across reasonable data perturbations. We propose a new variable importance framework that quantifies the importance of a variable across the set of all good models and is stable across the data distribution. Our framework is extremely flexible and can be integrated with most existing model classes and global variable importance metrics. We demonstrate through experiments that our framework recovers variable importance rankings for complex simulation setups where other methods fail. Further, we show that our framework accurately estimates the true importance of a variable for the underlying data distribution. We provide theoretical guarantees on the consistency and finite sample error rates for our estimator. Finally, we demonstrate its utility with a real-world case study exploring which genes are important for predicting HIV load in persons with HIV, highlighting an important gene that has not previously been studied in connection with HIV. Code is available at <a class="link-external link-https" href="https://github.com/jdonnelly36/Rashomon_Importance_Distribution" rel="external noopener nofollow">this https URL</a>.

Variable importance in regression models

The Importance of Variable Importance

Assessing Variable Importance for Best Subset Selection

Regression and Independence Based Variable Importance Measure.

A Computational Exploration of Emerging Methods of Variable Importance Estimation

Challenges in Variable Importance Ranking Under Correlation

The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importance

A simple approach for local and global variable importance in nonlinear regression models

Variable importance: Comparison of selectivity ratio and significance multivariate correlation for interpretation of latent‐variable regression models

Variable importance in binary regression trees and forests

Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Models

Reliable and scalable variable importance estimation via warm-start and early stopping

Model free variable importance for high dimensional data

Nonparametric Feature Impact and Importance

Lazy Estimation of Variable Importance for Large Neural Networks

Model-independent variable selection via the rule-based variable priority

Bayesian variable selection in linear regression models with instrumental variables

Enhancing Variable Importance in Random Forests: A Novel Application of Global Sensitivity Analysis

Interpreting Deep Neural Networks Through Variable Importance

A Notion of Feature Importance by Decorrelation and Detection of Trends by Random Forest Regression

Variable Importance Assessments and Backward Variable Selection for High-Dimensional Data