Abstract:Software robots have long been used in Robotic Process Automation (RPA) to automate mundane and repetitive computer tasks. With the advent of Large Language Models (LLMs) and their advanced reasoning capabilities, these agents are now able to handle more complex or previously unseen tasks. However, LLM-based automation techniques in recent literature frequently rely on HTML source code for input or application-specific API calls for actions, limiting their applicability to specific environments. We propose an LLM-based agent that mimics human behavior in solving computer tasks. It perceives its environment solely through screenshot images, which are then converted into text for an LLM to process. By leveraging the reasoning capability of the LLM, we eliminate the need for large-scale human demonstration data typically required for model training. The agent only executes keyboard and mouse operations on Graphical User Interface (GUI), removing the need for pre-provided APIs to function. To further enhance the agent's performance in this setting, we propose a novel prompting strategy called Context-Aware Action Planning (CAAP) prompting, which enables the agent to thoroughly examine the task context from multiple perspectives. Our agent achieves an average success rate of 94.5% on MiniWoB++ and an average task score of 62.3 on WebShop, outperforming all previous studies of agents that rely solely on screen images. This method demonstrates potential for broader applications, particularly for tasks requiring coordination across multiple applications on desktops or smartphones, marking a significant advancement in the field of automation agents. Codes and models are accessible at <a class="link-external link-https" href="https://github.com/caap-agent/caap-agent" rel="external noopener nofollow">this https URL</a>.

AutoAct: Automatic Agent Learning from Scratch for QA Via Self-Planning

AUTOACT: Automatic Agent Learning from Scratch via Self-Planning

KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents

AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML

AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning

Agents: An Open-source Framework for Autonomous Language Agents

DialogAgent: An Auto-engagement Agent for Code Question Answering Data Production

ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomy

You Only Look at Screens: Multimodal Chain-of-Action Agents

AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation

Ask-before-Plan: Proactive Language Agents for Real-World Planning

Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents

Towards Autonomous Agents: Adaptive-planning, Reasoning, and Acting in Language Models

XUAT-Copilot: Multi-Agent Collaborative System for Automated User Acceptance Testing with Large Language Model

AutoPrep: Natural Language Question-Aware Data Preparation with a Multi-Agent Framework

Executable Code Actions Elicit Better LLM Agents

CAAP: Context-Aware Action Planning Prompting to Solve Computer Tasks with Front-End UI Only

PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planning