Abstract:Large language models (LLMs) demonstrate impressive reasoning abilities, but translating reasoning into actions in the real world remains challenging. In particular, it remains unclear how to complete a given task provably within a minimum number of interactions with the external environment, e.g., through an internal mechanism of reasoning. To this end, we propose a principled framework with provable regret guarantees to orchestrate reasoning and acting, which we call "reason for future, act for now" (\texttt{RAFA}). Specifically, we design a prompt template for reasoning that learns from the memory buffer and plans a future trajectory over a long horizon ("reason for future"). At each step, the LLM agent takes the initial action of the planned trajectory ("act for now"), stores the collected feedback in the memory buffer, and reinvokes the reasoning routine to replan the future trajectory from the new state. The key idea is to cast reasoning in LLMs as learning and planning in Bayesian adaptive Markov decision processes (MDPs). Correspondingly, we prompt LLMs to form an updated posterior of the unknown environment from the memory buffer (learning) and generate an optimal trajectory for multiple future steps that maximizes a value function (planning). The learning and planning subroutines are performed in an "in-context" manner to emulate the actor-critic update for MDPs. Our theoretical analysis proves that the novel combination of long-term reasoning and short-term acting achieves a $\sqrt{T}$ regret. Here, $T$ denotes the number of online interactions. In particular, the regret bound highlights an intriguing interplay between the prior knowledge obtained through pretraining and the uncertainty reduction achieved by reasoning and acting. Our empirical validation shows that it outperforms various existing frameworks and achieves nearly perfect scores on a few benchmarks.

RA: A machine based rational agent, Part 1

Advantage Actor-Critic with Reasoner: Explaining the Agent's Behavior from an Exploratory Perspective.

Learning Robust Rule Representations for Abstract Reasoning via Internal Inferences

Automated Machine Learning, Bounded Rationality, and Rational Metareasoning

Relational-Grid-World: A Novel Relational Reasoning Environment and An Agent Model for Relational Information Extraction

ACT-R: A cognitive architecture for modeling cognition.

Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptions

Reason for Future, Act for Now: A Principled Architecture for Autonomous LLM Agents

(Ir)rationality in AI: State of the Art, Research Challenges and Open Questions

ERRA: An Embodied Representation and Reasoning Architecture for Long-horizon Language-conditioned Manipulation Tasks

Reinforcement Learning via Reasoning from Demonstration

Toward a Psychology of Deep Reinforcement Learning Agents Using a Cognitive Architecture

Abstract Visual Reasoning: An Algebraic Approach for Solving Raven's Progressive Matrices

AGENT: A Benchmark for Core Psychological Reasoning

Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency

Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning

Probabilistic Abduction for Visual Abstract Reasoning via Learning Rules in Vector-symbolic Architectures

Abstract Spatial-Temporal Reasoning Via Probabilistic Abduction and Execution

Rational Decision-Making Agent with Internalized Utility Judgment

Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals

Towards Generative Abstract Reasoning: Completing Raven's Progressive Matrix via Rule Abstraction and Selection