Abstract:The convenient access to copious multi-faceted data has encouraged machine learning researchers to reconsider correlation-based learning and embrace the opportunity of causality-based learning, i.e., causal machine learning (causal learning). Recent years have therefore witnessed great effort in developing causal learning algorithms aiming to help AI achieve human-level intelligence. Due to the lack-of ground-truth data, one of the biggest challenges in current causal learning research is algorithm evaluations. This largely impedes the cross-pollination of AI and causal inference, and hinders the two fields to benefit from the advances of the other. To bridge from conventional causal inference (i.e., based on statistical methods) to causal learning with big data (i.e., the intersection of causal inference and machine learning), in this survey, we review commonly-used datasets, evaluation methods, and measures for causal learning using an evaluation pipeline similar to conventional machine learning. We focus on the two fundamental causal-inference tasks and causality-aware machine learning tasks. Limitations of current evaluation procedures are also discussed. We then examine popular causal inference tools/packages and conclude with primary challenges and opportunities for benchmarking causal learning algorithms in the era of big data. The survey seeks to bring to the forefront the urgency of developing publicly available benchmarks and consensus-building standards for causal learning evaluation with observational data. In doing so, we hope to broaden the discussions and facilitate collaboration to advance the innovation and application of causal learning.

The Causal Chambers: Real Physical Systems as a Testbed for AI Methodology

Towards Understanding How Machines Can Learn Causal Overhypotheses

Introducing CausalBench: A Flexible Benchmark Framework for Causal Analysis and Machine Learning

Methods and tools for causal discovery and causal inference

Causal Inference and Counterfactual Prediction in Machine Learning for Actionable Healthcare

CausalTime: Realistically Generated Time-series for Benchmarking of Causal Discovery

Testing Causality in Scientific Modelling Software

CausalBench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Data

Using causal inference to avoid fallouts in data-driven parametric analysis: a case study in the architecture, engineering, and construction industry

$\texttt{causalAssembly}$: Generating Realistic Production Data for Benchmarking Causal Discovery

Computational Causal Inference

The Landscape of Causal Discovery Data: Grounding Causal Discovery in Real-World Applications

AI Assurance using Causal Inference: Application to Public Policy

CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning

Causal inference in AI education: A primer

Inferring physical laws by artificial intelligence based causal models

Evaluation Methods and Measures for Causal Learning Algorithms

Causal Deep Learning

An Introduction to Causal Inference Methods for Observational Human-Robot Interaction Research

CausalQuest: Collecting Natural Causal Questions for AI Agents

Experimental Evaluation of ROS-Causal in Real-World Human-Robot Spatial Interaction Scenarios