Abstract:Recent months have seen the emergence of a powerful new trend in which large language models (LLMs) are augmented to become autonomous language agents capable of performing objective oriented multi-step tasks on their own, rather than merely responding to queries from human users. Most existing language agents, however, are not optimized using environment-specific rewards. Although some agents enable iterative refinement through verbal feedback, they do not reason and plan in ways that are compatible with gradient-based learning from rewards. This paper introduces a principled framework for reinforcing large language agents by learning a retrospective model, which automatically tunes the language agent prompts from environment feedback through policy gradient. Specifically, our proposed agent architecture learns from rewards across multiple environments and tasks, for fine-tuning a pre-trained language model which refines the language agent prompt by summarizing the root cause of prior failed attempts and proposing action plans. Experimental results on various tasks demonstrate that the language agents improve over time and that our approach considerably outperforms baselines that do not properly leverage gradients from the environment. This demonstrates that using policy gradient optimization to improve language agents, for which we believe our work is one of the first, seems promising and can be applied to optimize other models in the agent architecture to enhance agent performances over time.

Special Topic: AI Algorithms and Cases: To Energize Digital Economy Gradient-free tuning for large language models competition: approaches, results, current challenges and future directions

Competition for Gradient-Free Tuning of Large Language Models: Approaches, Results, Current Challenges and Future Directions

Special Topic: AI Algorithms and Cases: To Energize Digital Economy The PSG challenge: towards comprehensive scene understanding

Special Topic: AI Algorithms and Cases : To Energize Digital Economy Automated object recognition in high-resolution optical remote sensing imagery

When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenario

Influential Language Data Selection via Gradient Trajectory Pursuit

BBTv2: Towards a Gradient-Free Future with Large Language Models

G-DIG: Towards Gradient-based Diverse and High-quality Instruction Data Selection for Machine Translation

Bringing Generative AI to Adaptive Learning in Education

Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competition

On the Challenges and Opportunities in Generative AI

CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agents

Exploring Edge-driven Collaborative Fine-tuning Towards Customized AIGC Services

CompeteAI: Understanding the Competition Dynamics of Large Language Model-based Agents

Competition and Diversity in Generative AI

Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization

Gradient-Mask Tuning Elevates the Upper Limits of LLM Performance

Learning Gradient Descent: Better Generalization and Longer Horizons

Engineering Exploration and Research on Large Language Models

Make Prompt-based Black-Box Tuning Colorful: Boosting Model Generalization from Three Orthogonal Perspectives

CompeteAI: Understanding the Competition Behaviors in Large Language Model-based Agents