Abstract:The performance of most causal effect estimators relies on accurate predictions of high-dimensional non-linear functions of the observed data. The remarkable flexibility of modern Machine Learning (ML) methods is perfectly suited to this task. However, data-driven hyperparameter tuning of ML methods requires effective model evaluation to avoid large errors in causal estimates, a task made more challenging because causal inference involves unavailable counterfactuals. Multiple performance-validation metrics have recently been proposed such that practitioners now not only have to make complex decisions about which causal estimators, ML learners and hyperparameters to choose, but also about which evaluation metric to use. This paper, motivated by unclear recommendations, investigates the interplay between the four different aspects of model evaluation for causal effect estimation. We develop a comprehensive experimental setup that involves many commonly used causal estimators, ML methods and evaluation approaches and apply it to four well-known causal inference benchmark datasets. Our results suggest that optimal hyperparameter tuning of ML learners is enough to reach state-of-the-art performance in effect estimation, regardless of estimators and learners. We conclude that most causal estimators are roughly equivalent in performance if tuned thoroughly enough. We also find hyperparameter tuning and model evaluation are much more important than causal estimators and ML methods. Finally, from the significant gap we find in estimation performance of popular evaluation metrics compared with optimal model selection choices, we call for more research into causal model evaluation to unlock the optimum performance not currently being delivered even by state-of-the-art procedures.

About evaluation metrics for contextual uplift modeling

Improving uplift model evaluation on RCT data

Optimising Individual-Treatment-Effect Using Bandits

Partial counterfactual identification and uplift modeling: theoretical results and real-world assessment

A Large Scale Benchmark for Individual Treatment Effect Prediction and Uplift Modeling

Causal Enhanced Uplift Model

Uplift vs. predictive modeling: a theoretical analysis

A Literature Survey and Experimental Evaluation of the State-of-the-Art in Uplift Modeling: A Stepping Stone Toward the Development of Prescriptive Analytics

Uplift Modeling with Multiple Treatments and General Response Types

Uplift Modeling: from Causal Inference to Personalization

Hyperparameter Tuning and Model Evaluation in Causal Effect Estimation

Adapting Neural Networks for Uplift Models

Enhancing Uplift Modeling in Multi-Treatment Marketing Campaigns: Leveraging Score Ranking and Calibration Techniques

Causal Inference Based Single-branch Ensemble Trees For Uplift Modeling

Uplift Modeling for Multiple Treatments with Cost Optimization

Post-Contextual-Bandit Inference

Entire Chain Uplift Modeling with Context-Enhanced Learning for Intelligent Marketing

A Twin Neural Model for Uplift

Weighted doubly robust learning: An uplift modeling technique for estimating mixed treatments' effect

High Precision Causal Model Evaluation with Conditional Randomization

Imbalance-Aware Uplift Modeling for Observational Data