Abstract:The causal capabilities of large language models (LLMs) are a matter of significant debate, with critical implications for the use of LLMs in societally impactful domains such as medicine, science, law, and policy. We conduct a "behavorial" study of LLMs to benchmark their capability in generating causal arguments. Across a wide range of tasks, we find that LLMs can generate text corresponding to correct causal arguments with high probability, surpassing the best-performing existing methods. Algorithms based on GPT-3.5 and 4 outperform existing algorithms on a pairwise causal discovery task (97%, 13 points gain), counterfactual reasoning task (92%, 20 points gain) and event causality (86% accuracy in determining necessary and sufficient causes in vignettes). We perform robustness checks across tasks and show that the capabilities cannot be explained by dataset memorization alone, especially since LLMs generalize to novel datasets that were created after the training cutoff date. That said, LLMs exhibit unpredictable failure modes, and we discuss the kinds of errors that may be improved and what are the fundamental limits of LLM-based answers. Overall, by operating on the text metadata, LLMs bring capabilities so far understood to be restricted to humans, such as using collected knowledge to generate causal graphs or identifying background causal context from natural language. As a result, LLMs may be used by human domain experts to save effort in setting up a causal analysis, one of the biggest impediments to the widespread adoption of causal methods. Given that LLMs ignore the actual data, our results also point to a fruitful research direction of developing algorithms that combine LLMs with existing causal techniques. Code and datasets are available at <a class="link-external link-https" href="https://github.com/py-why/pywhy-llm" rel="external noopener nofollow">this https URL</a>.

Causal Dataset Discovery with Large Language Models

Causal Reasoning and Large Language Models: Opening a New Frontier for Causality

Large Language Model for Causal Decision Making

From Query Tools to Causal Architects: Harnessing Large Language Models for Advanced Causal Discovery from Data

Evaluating Large Language Models for Causal Modeling

From Pre-training Corpora to Large Language Models: What Factors Influence LLM Performance in Causal Discovery Tasks?

Causal Inference with Large Language Model: A Survey

Large Language Models for Constrained-Based Causal Discovery

Causality for Large Language Models

Can Large Language Models Infer Causation from Correlation?

Counterfactual Causal Inference in Natural Language with Large Language Models

Integrating Large Language Models in Causal Discovery: A Statistical Causal Approach

Large Language Models and Causal Inference in Collaboration: A Comprehensive Survey

LLM4Causal: Democratized Causal Tools for Everyone via Large Language Model

Multi-Agent Causal Discovery Using Large Language Models

Discovery of the Hidden World with Large Language Models

Efficient Causal Graph Discovery Using Large Language Models

Cause and Effect: Can Large Language Models Truly Understand Causality?

Causal Structure Learning Supervised by Large Language Model

Causal Graph Discovery with Retrieval-Augmented Generation based Large Language Models

Can large language models build causal graphs?