Spatial Assisted Human-Drone Collaborative Navigation and Interaction through Immersive Mixed Reality

Luca Morando,Giuseppe Loianno
2024-04-07
Abstract:Aerial robots have the potential to play a crucial role in assisting humans with complex and dangerous tasks. Nevertheless, the future industry demands innovative solutions to streamline the interaction process between humans and drones to enable seamless collaboration and efficient co-working. In this paper, we present a novel tele-immersive framework that promotes cognitive and physical collaboration between humans and robots through Mixed Reality (MR). This framework incorporates a novel bi-directional spatial awareness and a multi-modal virtual-physical interaction approaches. The former seamlessly integrates the physical and virtual worlds, offering bidirectional egocentric and exocentric environmental representations. The latter, leveraging the proposed spatial representation, further enhances the collaboration combining a robot planning algorithm for obstacle avoidance with a variable admittance control. This allows users to issue commands based on virtual forces while maintaining compatibility with the environment map. We validate the proposed approach by performing several collaborative planning and exploration tasks involving a drone and an user equipped with a MR headset.
Robotics,Systems and Control
What problem does this paper attempt to address?
The paper attempts to address the issue of the lack of seamless collaboration and efficient cooperation between humans and drones in complex or dangerous tasks. Specifically, the paper proposes a new remote immersive framework that facilitates cognitive and physical collaboration between humans and drones through Mixed Reality (MR) technology. This framework includes a bidirectional spatial awareness concept that enables humans and drones to share spatial information of the environment. It also introduces a new virtual-physical interaction method that combines Variable Admittance Control (VAC) with obstacle avoidance planning algorithms, allowing users to input virtual external forces as robot commands through gesture recognition while ensuring compatibility with the environmental map. This framework aims to enhance mutual understanding and support in human-drone collaboration, making drones not just auxiliary tools but true partners in various tasks. Additionally, this work has been open-sourced for use and development by the research community.