OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning

Xiaoqiang Wang,Bang Liu

2024-10-25

Abstract:Large language models (LLMs) and large multimodal models (LMMs) have shown great potential in automating complex tasks like web browsing and gaming. However, their ability to generalize across diverse applications remains limited, hindering broader utility. To address this challenge, we present OSCAR: Operating System Control via state-Aware reasoning and Re-planning. OSCAR is a generalist agent designed to autonomously navigate and interact with various desktop and mobile applications through standardized controls, such as mouse and keyboard inputs, while processing screen images to fulfill user commands. OSCAR translates human instructions into executable Python code, enabling precise control over graphical user interfaces (GUIs). To enhance stability and adaptability, OSCAR operates as a state machine, equipped with error-handling mechanisms and dynamic task re-planning, allowing it to efficiently adjust to real-time feedback and exceptions. We demonstrate OSCAR's effectiveness through extensive experiments on diverse benchmarks across desktop and mobile platforms, where it transforms complex workflows into simple natural language commands, significantly boosting user productivity. Our code will be open-source upon publication.

Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve the ability of a generalist agent in different applications, enabling it to autonomously navigate and interact with various desktop and mobile applications according to users' natural language instructions. Although existing large - language models (LLMs) and large multi - modal models (LMMs) perform well in automating complex tasks such as web browsing and gaming, their generalization ability between different applications is limited, which restricts their wider application. To overcome this challenge, the paper proposes OSCAR (Operating System Control via State - Aware Reasoning and Re - planning), which is a generalist agent designed to autonomously navigate and interact with multiple applications through standardized controls (such as mouse and keyboard inputs).

The main features of OSCAR include:
1. **Unified control interface**: OSCAR can seamlessly operate standard input methods, such as the mouse and keyboard, to perform precise actions across multiple applications, such as mouse movement, clicking, scrolling, and using keyboard shortcuts.
2. **GUI grounding**: OSCAR needs to interpret the screen and accurately identify relevant elements, such as buttons, menus, or text fields. It utilizes the operating system window API to extract interactive elements and their spatial layout to enhance GUI understanding.
3. **Exploration - based simulation and re - planning**: Similar to the way humans navigate in unfamiliar software interfaces, OSCAR must have the ability to explore and dynamically adjust plans. This includes retrying actions, handling exceptions (such as software crashes), and adjusting strategies according to real - time feedback from the system.

Through these mechanisms, OSCAR can efficiently handle user requests in a dynamic environment, adapt to environmental changes, and make adjustments when execution fails, thereby significantly improving user productivity. The paper verifies the effectiveness and generalization ability of OSCAR through experiments on multiple benchmarks, especially outstanding in complex multi - application tasks.