LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theory

Jillian Ross,Yoon Kim,Andrew W. Lo

2024-08-06

Abstract:Humans are not homo economicus (i.e., rational economic beings). As humans, we exhibit systematic behavioral biases such as loss aversion, anchoring, framing, etc., which lead us to make suboptimal economic decisions. Insofar as such biases may be embedded in text data on which large language models (LLMs) are trained, to what extent are LLMs prone to the same behavioral biases? Understanding these biases in LLMs is crucial for deploying LLMs to support human decision-making. We propose utility theory-a paradigm at the core of modern economic theory-as an approach to evaluate the economic biases of LLMs. Utility theory enables the quantification and comparison of economic behavior against benchmarks such as perfect rationality or human behavior. To demonstrate our approach, we quantify and compare the economic behavior of a variety of open- and closed-source LLMs. We find that the economic behavior of current LLMs is neither entirely human-like nor entirely economicus-like. We also find that most current LLMs struggle to maintain consistent economic behavior across settings. Finally, we illustrate how our approach can measure the effect of interventions such as prompting on economic biases.

Computation and Language

What problem does this paper attempt to address?

The paper primarily explores the issue of behavioral biases in large language models (LLMs) in economic decision-making. Specifically, the paper evaluates the performance of these models in three aspects: inequality aversion, risk and loss aversion, and time discounting by applying experimental games from behavioral economics to LLMs. The study finds that current LLMs exhibit behaviors in these three aspects that are neither like completely rational human economic agents (homo economicus) nor entirely consistent with humans. For example:

1. **Inequality Aversion**: LLMs generally exhibit weaker self-inequality aversion but show stronger inequality aversion towards others.
2. **Risk and Loss Aversion**: The degree of risk aversion in LLMs is similar to that of humans, but when facing potential losses, their behavior is more irrational than that of humans.
3. **Time Discounting**: LLMs exhibit a stronger preference for immediate rewards over delayed gratification compared to humans in terms of time discounting.

Additionally, the paper explores methods to intervene in the economic behavior of LLMs through prompting techniques, including direct prompts, chain-of-thought prompts, and single and double example prompts. The results show that although some prompting methods can change the behavior of LLMs to a certain extent, the effects are not stable and have limitations. Finally, the paper points out directions for further research, including understanding the consistency of LLMs' behavior in different economic environments and how to better adjust their economic decision-making capabilities to suit practical applications.