Abstract:The causal capabilities of large language models (LLMs) are a matter of significant debate, with critical implications for the use of LLMs in societally impactful domains such as medicine, science, law, and policy. We conduct a "behavorial" study of LLMs to benchmark their capability in generating causal arguments. Across a wide range of tasks, we find that LLMs can generate text corresponding to correct causal arguments with high probability, surpassing the best-performing existing methods. Algorithms based on GPT-3.5 and 4 outperform existing algorithms on a pairwise causal discovery task (97%, 13 points gain), counterfactual reasoning task (92%, 20 points gain) and event causality (86% accuracy in determining necessary and sufficient causes in vignettes). We perform robustness checks across tasks and show that the capabilities cannot be explained by dataset memorization alone, especially since LLMs generalize to novel datasets that were created after the training cutoff date. That said, LLMs exhibit unpredictable failure modes, and we discuss the kinds of errors that may be improved and what are the fundamental limits of LLM-based answers. Overall, by operating on the text metadata, LLMs bring capabilities so far understood to be restricted to humans, such as using collected knowledge to generate causal graphs or identifying background causal context from natural language. As a result, LLMs may be used by human domain experts to save effort in setting up a causal analysis, one of the biggest impediments to the widespread adoption of causal methods. Given that LLMs ignore the actual data, our results also point to a fruitful research direction of developing algorithms that combine LLMs with existing causal techniques. Code and datasets are available at <a class="link-external link-https" href="https://github.com/py-why/pywhy-llm" rel="external noopener nofollow">this https URL</a>.

Causal Graph Discovery with Retrieval-Augmented Generation based Large Language Models

Efficient Causal Graph Discovery Using Large Language Models

Large Language Models are Effective Priors for Causal Graph Discovery

Large Language Models for Constrained-Based Causal Discovery

From Query Tools to Causal Architects: Harnessing Large Language Models for Advanced Causal Discovery from Data

Can large language models build causal graphs?

Large Language Models and Causal Inference in Collaboration: A Comprehensive Survey

CausalGraph2LLM: Evaluating LLMs for Causal Queries

Counterfactual Causal Inference in Natural Language with Large Language Models

Causal Dataset Discovery with Large Language Models

Automating psychological hypothesis generation with AI: when large language models meet causal graph

Causal Reasoning and Large Language Models: Opening a New Frontier for Causality

Causal Inference Using LLM-Guided Discovery

Multi-Agent Causal Discovery Using Large Language Models

Causal Reasoning in Large Language Models: A Knowledge Graph Approach

Causality for Large Language Models

CLEAR: Can Language Models Really Understand Causal Graphs?

Large Language Model for Causal Decision Making

Causal Structure Learning Supervised by Large Language Model

Improving Causal Reasoning in Large Language Models: A Survey

From Pre-training Corpora to Large Language Models: What Factors Influence LLM Performance in Causal Discovery Tasks?