Reflexion: Language Agents with Verbal Reinforcement Learning

Noah Shinn,Federico Cassano,Edward Berman,Ashwin Gopinath,Karthik Narasimhan,Shunyu Yao

2023-10-10

Abstract:Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.

Artificial Intelligence,Computation and Language,Machine Learning

What problem does this paper attempt to address?

The problem this paper attempts to address is: How to enable large language models (LLMs) to learn quickly and efficiently from trial and error when interacting with external environments, without requiring the large number of training samples and expensive model fine-tuning needed by traditional reinforcement learning methods. Specifically, the paper proposes a new framework called Reflexion, which enhances the learning ability of language agents through language feedback rather than weight updates. Reflexion agents verbally reflect on task feedback signals and store their own reflective text in a situational memory buffer to make better decisions in subsequent trials. This framework is applicable to various types of feedback signals (numerical or free-form language) and sources (external or internal simulation), and has achieved significant improvements across different tasks (sequential decision-making, programming, language reasoning). For example, in the HumanEval programming benchmark, Reflexion achieved a 91% pass@1 accuracy, surpassing the previous state-of-the-art GPT-4 which achieved 80%.