Can large language models explore in-context?

Akshay Krishnamurthy,Keegan Harris,Dylan J. Foster,Cyril Zhang,Aleksandrs Slivkins
2024-10-29
Abstract:We investigate the extent to which contemporary Large Language Models (LLMs) can engage in exploration, a core capability in reinforcement learning and decision making. We focus on native performance of existing LLMs, without training interventions. We deploy LLMs as agents in simple multi-armed bandit environments, specifying the environment description and interaction history entirely in-context, i.e., within the LLM prompt. We experiment with GPT-3.5, GPT-4, and Llama2, using a variety of prompt designs, and find that the models do not robustly engage in exploration without substantial interventions: i) Across all of our experiments, only one configuration resulted in satisfactory exploratory behavior: GPT-4 with chain-of-thought reasoning and an externally summarized interaction history, presented as sufficient statistics; ii) All other configurations did not result in robust exploratory behavior, including those with chain-of-thought reasoning but unsummarized history. Although these findings can be interpreted positively, they suggest that external summarization -- which may not be possible in more complex settings -- is important for obtaining desirable behavior from LLM agents. We conclude that non-trivial algorithmic interventions, such as fine-tuning or dataset curation, may be required to empower LLM-based decision making agents in complex settings.
Machine Learning,Artificial Intelligence,Computation and Language
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: Can large language models (LLMs) explore in context without additional training interventions, that is, given the environmental description and interaction history as prompts, can LLMs, like reinforcement - learning agents, reduce uncertainty and optimize the decision - making process by trying different options? Specifically, the authors focus on the performance of LLMs in the multi - armed bandit (MAB) environment, which is a classic reinforcement - learning problem involving how to make a trade - off between exploration (trying new options to obtain more information) and exploitation (selecting the known best option). The researchers hope to experimentally verify whether existing LLMs can perform well in such tasks requiring exploration ability without special training or adjustment of the model. The main contribution of the paper lies in systematically evaluating the exploration behaviors of different LLMs (such as GPT - 3.5, GPT - 4 and Llama2) in the multi - armed bandit task, using a variety of different prompt designs, and analyzing which factors will affect the exploration ability of LLMs. The research results show that only under specific configurations (for example, when using GPT - 4 in combination with enhanced prompt designs), can LLMs effectively explore, while in most other configurations, the performance of LLMs is not ideal and exploration failures often occur. These findings are of great significance for understanding the application potential of LLMs in complex decision - making tasks.