Abstract:The causal capabilities of large language models (LLMs) are a matter of significant debate, with critical implications for the use of LLMs in societally impactful domains such as medicine, science, law, and policy. We conduct a "behavorial" study of LLMs to benchmark their capability in generating causal arguments. Across a wide range of tasks, we find that LLMs can generate text corresponding to correct causal arguments with high probability, surpassing the best-performing existing methods. Algorithms based on GPT-3.5 and 4 outperform existing algorithms on a pairwise causal discovery task (97%, 13 points gain), counterfactual reasoning task (92%, 20 points gain) and event causality (86% accuracy in determining necessary and sufficient causes in vignettes). We perform robustness checks across tasks and show that the capabilities cannot be explained by dataset memorization alone, especially since LLMs generalize to novel datasets that were created after the training cutoff date. That said, LLMs exhibit unpredictable failure modes, and we discuss the kinds of errors that may be improved and what are the fundamental limits of LLM-based answers. Overall, by operating on the text metadata, LLMs bring capabilities so far understood to be restricted to humans, such as using collected knowledge to generate causal graphs or identifying background causal context from natural language. As a result, LLMs may be used by human domain experts to save effort in setting up a causal analysis, one of the biggest impediments to the widespread adoption of causal methods. Given that LLMs ignore the actual data, our results also point to a fruitful research direction of developing algorithms that combine LLMs with existing causal techniques. Code and datasets are available at <a class="link-external link-https" href="https://github.com/py-why/pywhy-llm" rel="external noopener nofollow">this https URL</a>.

Embracing Background Knowledge in the Analysis of Actual Causality: An Answer Set Programming Approach

Causal Inference and Causal Explanation with Background Knowledge

Towards a General Framework for Actual Causation Using CP-logic

Combining Probabilistic, Causal, and Normative Reasoning in CP-logic

New Rules for Causal Identification with Background Knowledge

Causality Measures and Analysis: A Rough Set Framework.

Local Causal Discovery with Background Knowledge

CausalKG: Causal Knowledge Graph Explainability using interventional and counterfactual reasoning

CauseKG: A Framework Enhancing Causal Inference With Implicit Knowledge Deduced From Knowledge Graphs

Sound and Complete Causal Identification with Latent Variables Given Local Background Knowledge.

Learning Causality and Causality-Related Learning: Some Recent Progress.

On the Representation of Causal Background Knowledge and its Applications in Causal Inference

Language Agents Meet Causality -- Bridging LLMs and Causal World Models

A formalism for causal explanations with an Answer Set Programming translation

Argumentative Causal Discovery

Causal Deep Learning

Whittemore: An embedded domain specific language for causal programming

Cause and Effect: Can Large Language Models Truly Understand Causality?

Causal discovery using dynamically requested knowledge

Causal Reasoning and Large Language Models: Opening a New Frontier for Causality