Abstract:The causal capabilities of large language models (LLMs) are a matter of significant debate, with critical implications for the use of LLMs in societally impactful domains such as medicine, science, law, and policy. We conduct a "behavorial" study of LLMs to benchmark their capability in generating causal arguments. Across a wide range of tasks, we find that LLMs can generate text corresponding to correct causal arguments with high probability, surpassing the best-performing existing methods. Algorithms based on GPT-3.5 and 4 outperform existing algorithms on a pairwise causal discovery task (97%, 13 points gain), counterfactual reasoning task (92%, 20 points gain) and event causality (86% accuracy in determining necessary and sufficient causes in vignettes). We perform robustness checks across tasks and show that the capabilities cannot be explained by dataset memorization alone, especially since LLMs generalize to novel datasets that were created after the training cutoff date. That said, LLMs exhibit unpredictable failure modes, and we discuss the kinds of errors that may be improved and what are the fundamental limits of LLM-based answers. Overall, by operating on the text metadata, LLMs bring capabilities so far understood to be restricted to humans, such as using collected knowledge to generate causal graphs or identifying background causal context from natural language. As a result, LLMs may be used by human domain experts to save effort in setting up a causal analysis, one of the biggest impediments to the widespread adoption of causal methods. Given that LLMs ignore the actual data, our results also point to a fruitful research direction of developing algorithms that combine LLMs with existing causal techniques. Code and datasets are available at <a class="link-external link-https" href="https://github.com/py-why/pywhy-llm" rel="external noopener nofollow">this https URL</a>.

Applying Large Language Models for Causal Structure Learning in Non Small Cell Lung Cancer

From Query Tools to Causal Architects: Harnessing Large Language Models for Advanced Causal Discovery from Data

Large Language Models as Co-Pilots for Causal Inference in Medical Studies

Can large language models build causal graphs?

Large Language Models for Constrained-Based Causal Discovery

Causal Inference with Large Language Model: A Survey

Causal Reasoning and Large Language Models: Opening a New Frontier for Causality

Large Language Models and Causal Inference in Collaboration: A Comprehensive Survey

Causal Structure Learning Supervised by Large Language Model

Causal Dataset Discovery with Large Language Models

Large Language Models are Effective Priors for Causal Graph Discovery

Counterfactual Causal Inference in Natural Language with Large Language Models

From Pre-training Corpora to Large Language Models: What Factors Influence LLM Performance in Causal Discovery Tasks?

Causality extraction from medical text using Large Language Models (LLMs)

Efficient Causal Graph Discovery Using Large Language Models

Causality for Large Language Models

Cause and Effect: Can Large Language Models Truly Understand Causality?

Large Language Model for Causal Decision Making

Evaluating Large Language Models for Causal Modeling

Causal Graph Discovery with Retrieval-Augmented Generation based Large Language Models

Deep Learning Causal Attributions of Breast Cancer