Abstract:State-of-the-art multimodal web agents, powered by Multimodal Large Language Models (MLLMs), can autonomously execute many web tasks by processing user instructions and interacting with graphical user interfaces (GUIs). Current strategies for building web agents rely on (i) the generalizability of underlying MLLMs and their steerability via prompting, and (ii) large-scale fine-tuning of MLLMs on web-related tasks. However, web agents still struggle to automate tasks on unseen websites and domains, limiting their applicability to enterprise-specific and proprietary platforms. Beyond generalization from large-scale pre-training and fine-tuning, we propose building agents for few-shot adaptability using human demonstrations. We introduce the AdaptAgent framework that enables both proprietary and open-weights multimodal web agents to adapt to new websites and domains using few human demonstrations (up to 2). Our experiments on two popular benchmarks -- Mind2Web & VisualWebArena -- show that using in-context demonstrations (for proprietary models) or meta-adaptation demonstrations (for meta-learned open-weights models) boosts task success rate by 3.36% to 7.21% over non-adapted state-of-the-art models, corresponding to a relative increase of 21.03% to 65.75%. Furthermore, our additional analyses (a) show the effectiveness of multimodal demonstrations over text-only ones, (b) shed light on the influence of different data selection strategies during meta-learning on the generalization of the agent, and (c) demonstrate the effect of number of few-shot examples on the web agent's success rate. Overall, our results unlock a complementary axis for developing widely applicable multimodal web agents beyond large-scale pre-training and fine-tuning, emphasizing few-shot adaptability.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that, in the current state, multi - modal web agents have limited generalization ability when facing unseen websites and domains, resulting in restricted applications on enterprise - specific and proprietary platforms. Although the capabilities of these agents can be improved through large - scale pre - training and fine - tuning, they are still difficult to automate tasks on unseen websites and domains. Therefore, the paper proposes a new framework - AdaptAgent, which aims to use a small number of human demonstrations (i.e., 1 to 2 examples) to enhance the adaptability of multi - modal web agents to new websites and domains. Specifically, the AdaptAgent framework enables proprietary and open - weight multi - modal web agents to quickly adapt to new environments through a small number of human demonstrations, thereby increasing the task success rate. The main contributions of the paper are as follows: 1. Propose the AdaptAgent framework for enabling state - of - the - art multi - modal web agents to learn from a small number of human demonstrations. This framework uses in - context learning (ICL) to adapt to proprietary multi - modal large language models (MLLMs), such as GPT - 4o, and uses meta - learning to adapt to open - weight MLLMs, such as CogAgent. 2. Demonstrate the effectiveness of this method through a large number of experiments on two widely adopted benchmarks - Mind2Web and VisualWebArena, and significantly improve the task success rate on unseen websites and domains with only 1 or 2 multi - modal demonstrations. 3. Conduct additional analysis and provide specific suggestions for future research and practice, including that multi - modal context demonstrations are more effective than pure - text demonstrations, different data selection strategies have an impact on the generalization ability of agents after meta - learning, and more demonstrations, while helpful for improving performance, also bring higher computational costs and the problem of diminishing returns. In conclusion, by proposing the AdaptAgent framework, the paper has opened up a new direction, that is, in addition to relying on large - scale pre - training and fine - tuning, the generalization ability of multi - modal web agents can also be improved through a small number of human demonstrations.

AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations

A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis

VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

Zero-shot Task Adaptation using Natural Language

WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

MMInA: Benchmarking Multihop Multimodal Internet Agents

OpenWebAgent: An Open Toolkit to Enable Web Agents on Large Language Models

Multimodal Web Navigation with Instruction-Finetuned Foundation Models

NNetscape Navigator: Complex Demonstrations for Web Agents Without a Demonstrator

OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization

ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data

AdaDemo: Data-Efficient Demonstration Expansion for Generalist Robotic Agent

Mind2Web: Towards a Generalist Agent for the Web

WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents

AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents

Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale

Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass

MARLUI: Multi-Agent Reinforcement Learning for Adaptive UIs

Domain Adaptation of Visual Policies with a Single Demonstration

Making the Most of What You Have: Adapting Pre-trained Visual Language Models in the Low-data Regime

OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web