Abstract:Recent advancements in language models have demonstrated remarkable improvements in various natural language processing (NLP) tasks such as web navigation. Supervised learning (SL) approaches have achieved impressive performance while utilizing significantly less training data compared to previous methods. However, these SL-based models fall short when compared to reinforcement learning (RL) approaches, which have shown superior results. In this paper, we propose a novel approach that combines SL and RL techniques over the MiniWoB benchmark to leverage the strengths of both methods. We also address a critical limitation in previous models' understanding of HTML content, revealing a tendency to memorize target elements rather than comprehend the underlying structure. To rectify this, we propose methods to enhance true understanding and present a new baseline of results. Our experiments demonstrate that our approach outperforms previous SL methods on certain tasks using less data and narrows the performance gap with RL models, achieving 43.58\% average accuracy in SL and 36.69\% when combined with a multimodal RL approach. This study sets a new direction for future web navigation and offers insights into the limitations and potential of language modeling for computer tasks.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the limitations of existing large - language models (LLMs) when performing Web tasks, especially in understanding and processing HTML content. Specifically, the paper focuses on the following key issues: 1. **Insufficient generalization ability of the model**: Existing LLMs perform well on specific Web navigation benchmarks (such as Miniwob++), but they are often sensitive to environmental changes and cannot adapt well to new or slightly changed task environments. This indicates that these models may just remember the specific element distribution rather than truly understand the HTML structure. 2. **Over - fitting problem**: Many previous studies have shown that models tend to over - fit their benchmark datasets during the training process, resulting in poor performance when encountering new tasks or environments. For example, a model may remember the specific location of the target element instead of learning how to parse and understand the web page structure. 3. **Combining the advantages of supervised learning (SL) and reinforcement learning (RL)**: Although SL methods can achieve good performance with less data, their performance still lags behind that of RL. The paper proposes a new method that combines SL and RL, aiming to utilize the advantages of both to improve the performance of the model in Web tasks. 4. **Challenges of multi - modal representation**: Combining visual input with text input for Web navigation is a complex problem. The paper explores how to enhance the model's understanding ability through multi - modal learning and proposes improved methods. ### Core contributions of the paper To address the above problems, the paper makes the following main contributions: - **Proposing a new method that combines SL and RL**: By combining SL and RL techniques on the Miniwob++ benchmark, the advantages of both are fully utilized, and the performance of the model is improved. - **Enhancing the understanding of HTML content**: The paper proposes some methods to correct the problem that the model tends to remember the target element rather than understand the HTML structure, thereby improving the model's true understanding ability. - **Setting more accurate benchmark results**: By improving the Miniwob++ benchmark, the paper provides more reliable and more practical performance evaluations, revealing some limitations in previous work. - **Introducing hierarchical planning ability**: By designing a model with hierarchical planning ability, the paper shows how to achieve better performance in complex Web tasks. In summary, this paper aims to improve the generalization ability and true understanding level of LLMs in Web navigation tasks by improving existing methods, providing new directions and ideas for future research.

Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning

WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning

Large Language Models Can Self-Improve At Web Agent Tasks

A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis

AutoWebGLM: A Large Language Model-based Web Navigating Agent

Search Beyond Queries: Training Smaller Language Models for Web Interactions via Reinforcement Learning

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation

Multimodal Web Navigation with Instruction-Finetuned Foundation Models

Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration

ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data

Language-Conditioned Offline RL for Multi-Robot Navigation

WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents

VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

WebVLN: Vision-and-Language Navigation on Websites

Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents

Tree Search for Language Model Agents

WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

Large Language Model Powered Agents in the Web

Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Web

AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?