Abstract:Multi-agent embodied tasks have recently been studied in complex indoor visual environments. Collaboration among multiple agents can improve work efficiency and has significant practical value. However, most of the existing research focuses on homogeneous multi-agent tasks. Compared with homogeneous agents, heterogeneous agents can leverage their different capabilities to allocate corresponding sub-tasks and cooperate to complete complex tasks. Heterogeneous multi-agent tasks are common in real-world scenarios, and the collaboration strategy among heterogeneous agents is a challenging and important problem to be solved. To study collaboration among heterogeneous agents, we propose the heterogeneous multi-agent tidying-up task, in which multiple heterogeneous agents with different capabilities collaborate with each other to detect misplaced objects and place them in reasonable locations. This is a demanding task since it requires agents to make the best use of their different capabilities to conduct reasonable task planning and complete the whole task. To solve this task, we build a heterogeneous multi-agent tidying-up benchmark dataset in a large number of houses with multiple rooms based on ProcTHOR-10K. We propose the hierarchical decision model based on misplaced object detection, reasonable receptacle prediction, as well as the handshake-based group communication mechanism. Extensive experiments are conducted to demonstrate the effectiveness of the proposed model. The project's website and videos of experiments can be found at <a class="link-external link-https" href="https://hetercol.github.io/" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### Problems Addressed by the Paper The paper primarily focuses on the collaboration strategies among heterogeneous multi-agents in complex indoor visual environments. Specifically, the paper proposes a heterogeneous multi-agent tidying-up task, where multiple agents with different capabilities need to cooperate to detect misplaced items and place them in appropriate locations. #### Main Issues: 1. **Collaboration of Heterogeneous Agents**: Compared to traditional homogeneous multi-agent tasks, heterogeneous agents can leverage their different capabilities to allocate corresponding subtasks and collaboratively complete complex tasks. Most current research focuses on homogeneous multi-agent tasks, while collaboration among heterogeneous agents is a more challenging and practically valuable problem. 2. **Environmental Perception and Task Planning**: Agents need to complete the tidying-up task in complex multi-room scenarios through reasonable task planning and collaboration. This requires agents to fully utilize their perception, navigation, and manipulation capabilities. 3. **Communication Mechanism**: The paper proposes a handshake-based group communication mechanism to achieve effective collaboration among heterogeneous agents. This mechanism supports not only intra-group communication but also inter-group communication, thereby improving task completion efficiency. ### Summary The core of the paper is to address how heterogeneous multi-agents can effectively collaborate to complete tidying-up tasks in complex indoor environments. Particularly in multi-room scenarios, agents need to utilize their respective capabilities for reasonable task allocation and planning. Additionally, the paper introduces a new benchmark dataset and validates the effectiveness of the proposed heterogeneous multi-agent collaboration framework.

Heterogeneous Embodied Multi-Agent Collaboration

Leveraging Large Language Model for Heterogeneous Ad Hoc Teamwork Collaboration

Heterogeneous multi-agent task allocation based on graph neural network ant colony optimization algorithms

COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models

Embodied Multi-Agent Task Planning from Ambiguous Instruction

$\textbf{EMOS}$: $\textbf{E}$mbodiment-aware Heterogeneous $\textbf{M}$ulti-robot $\textbf{O}$perating $\textbf{S}$ystem with LLM Agents

Enhancing Collaboration in Heterogeneous Multiagent Systems Through Communication Complementary Graph

Towards Practical Multi-Robot Hybrid Tasks Allocation for Autonomous Cleaning

Effectively Rearranging Heterogeneous Objects on Cluttered Tabletops

Combinatorial-hybrid Optimization for Multi-agent Systems under Collaborative Tasks

Heterogeneous Multi-Task Assignment in Mobile Crowdsensing Using Spatiotemporal Correlation

Task Planning for Object Rearrangement in Multi-room Environments

Cooperative Heterogeneous Multi-Robot Systems

CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamics

Aggregation-based dual heterogeneous task allocation in spatial crowdsourcing

Exploiting Heterogeneous Robotic Systems in Cooperative Missions

Multiagent Cooperation Based Entertainment Robot.

PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks

An Extensible Framework for Open Heterogeneous Collaborative Perception

Multi-agent mobile crowdsensing by pervasive machines: a robust task allocation approach

Human-Robot Collaboration Using Monte Carlo Tree Search