Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents

Yifei Zhou,Qianlan Yang,Kaixiang Lin,Min Bai,Xiong Zhou,Yu-Xiong Wang,Sergey Levine,Erran Li

2024-12-18

Abstract:The vision of a broadly capable and goal-directed agent, such as an Internet-browsing agent in the digital world and a household humanoid in the physical world, has rapidly advanced, thanks to the generalization capability of foundation models. Such a generalist agent needs to have a large and diverse skill repertoire, such as finding directions between two travel locations and buying specific items from the Internet. If each skill needs to be specified manually through a fixed set of human-annotated instructions, the agent's skill repertoire will necessarily be limited due to the quantity and diversity of human-annotated instructions. In this work, we address this challenge by proposing Proposer-Agent-Evaluator, an effective learning system that enables foundation model agents to autonomously discover and practice skills in the wild. At the heart of PAE is a context-aware task proposer that autonomously proposes tasks for the agent to practice with context information of the environment such as user demos or even just the name of the website itself for Internet-browsing agents. Then, the agent policy attempts those tasks with thoughts and actual grounded operations in the real world with resulting trajectories evaluated by an autonomous VLM-based success evaluator. The success evaluation serves as the reward signal for the agent to refine its policies through RL. We validate PAE on challenging vision-based web navigation, using both real-world and self-hosted websites from WebVoyager and <a class="link-external link-http" href="http://WebArena.To" rel="external noopener nofollow">this http URL</a> the best of our knowledge, this work represents the first effective learning system to apply autonomous task proposal with RL for agents that generalizes real-world human-annotated benchmarks with SOTA performances. Our open-source checkpoints and code can be found in <a class="link-external link-https" href="https://yanqval.github.io/PAE/" rel="external noopener nofollow">this https URL</a>

Machine Learning,Artificial Intelligence,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to enable agents based on foundation models to autonomously discover and practice skills, so as to effectively deal with various unseen tasks and environments in the real world without human supervision. Specifically, the paper proposes solutions to the following challenges:

1. **Limitations of the Skill Library**: If each skill needs to be manually specified through fixed human - annotated instructions, then the agent's skill library will be limited and cannot cover the wide and diverse tasks required in reality.

2. **Cost and Scalability of Human Annotation**: Generating high - quality human - annotated task templates is costly and difficult to scale up on a large scale, resulting in the agent being unable to adapt to the complexity and diversity of the real world.

3. **Insufficient Generalization Ability**: Agents trained with a small number of predefined task templates may encounter out - of - distribution problems when facing unseen tasks or environments, leading to performance degradation.

To solve these problems, the paper proposes the Proposer - Agent - Evaluator (PAE) framework, aiming to enable agents to autonomously discover and practice useful skills, thereby improving their zero - shot generalization ability in unseen tasks and environments. The PAE system achieves this goal through the following three main components:

- **Proposer**: Propose tasks autonomously based on environmental context information (such as user demonstrations or website names) to ensure the diversity and feasibility of tasks.
- **Agent Policy**: Try to execute the proposed tasks and interact with the environment through actual operations.
- **Evaluator**: Evaluate the success or failure of the task based on the final result and provide a reward signal for reinforcement learning (RL) to optimize the agent policy.

Through this design, the PAE framework not only reduces the dependence on human annotation but also significantly improves the performance of agents in unseen tasks and environments. Experimental results show that PAE achieves significantly better results than existing methods in multiple benchmark tests, especially in terms of zero - shot generalization ability.

### Main Contributions

1. **Autonomous Skill Discovery**: Proposed an effective learning system PAE, enabling agents based on foundation models to autonomously discover new skills without human supervision.
2. **Zero - Shot Generalization Ability**: Demonstrated that these autonomously discovered skills can be effectively applied to unseen real - world human - annotated tasks, achieving zero - shot generalization.
3. **Extensive Experimental Verification**: Conducted experiments in multiple challenging visual navigation environments to verify the effectiveness and superiority of PAE, especially performing well in benchmark tests such as WebVoyager and WebArena.

In summary, the main goal of this paper is to solve the problems of the limited agent skill library and insufficient generalization ability through the PAE framework, thereby achieving more extensive and flexible applications.