MMAC-Copilot: Multi-modal Agent Collaboration Operating System Copilot

Zirui Song,Yaohang Li,Meng Fang,Zhenhao Chen,Zecheng Shi,Yuan Huang,Ling Chen

2024-05-04

Abstract:Autonomous virtual agents are often limited by their singular mode of interaction with real-world environments, restricting their versatility. To address this, we propose the Multi-Modal Agent Collaboration framework (MMAC-Copilot), a framework utilizes the collective expertise of diverse agents to enhance interaction ability with operating systems. The framework introduces a team collaboration chain, enabling each participating agent to contribute insights based on their specific domain knowledge, effectively reducing the hallucination associated with knowledge domain gaps. To evaluate the performance of MMAC-Copilot, we conducted experiments using both the GAIA benchmark and our newly introduced Visual Interaction Benchmark (VIBench). VIBench focuses on non-API-interactable applications across various domains, including 3D gaming, recreation, and office scenarios. MMAC-Copilot achieved exceptional performance on GAIA, with an average improvement of 6.8\% over existing leading systems. Furthermore, it demonstrated remarkable capability on VIBench, particularly in managing various methods of interaction within systems and applications. These results underscore MMAC-Copilot's potential in advancing the field of autonomous virtual agents through its innovative approach to agent collaboration.

Artificial Intelligence,Human-Computer Interaction

What problem does this paper attempt to address?

The paper aims to address the limitations of autonomous virtual agents when interacting with real-world environments. Specifically, it proposes a Multimodal Agent Collaboration Framework (MMAC-Copilot) designed to enhance interaction capabilities with operating systems by leveraging the expertise of different agents. The framework introduces a team collaboration chain, enabling each participating agent to contribute insights based on its specific domain knowledge, thereby effectively reducing hallucinations caused by knowledge domain gaps.

The main contributions of the paper include:
1. Proposing the MMAC-Copilot framework, which enhances the interaction capabilities of autonomous virtual agents with operating systems through multimodal task processing.
2. Introducing a team collaboration chain mechanism that allows participating agents to adjust the initial plan based on their expertise, mitigating incorrect planning issues caused by LLM hallucinations.
3. Developing a new Visual Interaction Benchmark (VIBench) specifically designed to evaluate system performance in non-API-interactive applications, covering diverse fields such as 3D games, entertainment, and office scenarios.

Experimental results show that MMAC-Copilot performs excellently in the GAIA benchmark and demonstrates outstanding adaptability and accuracy in the newly proposed VIBench benchmark, particularly excelling in handling various interaction methods across different systems and applications. These results highlight the potential of MMAC-Copilot in its innovative agent collaboration approach.