SPIN: Simultaneous Perception, Interaction and Navigation

Shagun Uppal,Ananye Agarwal,Haoyu Xiong,Kenneth Shaw,Deepak Pathak

2024-05-14

Abstract:While there has been remarkable progress recently in the fields of manipulation and locomotion, mobile manipulation remains a long-standing challenge. Compared to locomotion or static manipulation, a mobile system must make a diverse range of long-horizon tasks feasible in unstructured and dynamic environments. While the applications are broad and interesting, there are a plethora of challenges in developing these systems such as coordination between the base and arm, reliance on onboard perception for perceiving and interacting with the environment, and most importantly, simultaneously integrating all these parts together. Prior works approach the problem using disentangled modular skills for mobility and manipulation that are trivially tied together. This causes several limitations such as compounding errors, delays in decision-making, and no whole-body coordination. In this work, we present a reactive mobile manipulation framework that uses an active visual system to consciously perceive and react to its environment. Similar to how humans leverage whole-body and hand-eye coordination, we develop a mobile manipulator that exploits its ability to move and see, more specifically -- to move in order to see and to see in order to move. This allows it to not only move around and interact with its environment but also, choose "when" to perceive "what" using an active visual system. We observe that such an agent learns to navigate around complex cluttered scenarios while displaying agile whole-body coordination using only ego-vision without needing to create environment maps. Results visualizations and videos at

Robotics,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning,Systems and Control

What problem does this paper attempt to address?

The paper attempts to address the problem of achieving simultaneous perception, interaction, and navigation (SPIN) with a mobile manipulator in cluttered and unstructured environments. Specifically, the paper focuses on how to enable robots to perform multiple long-term tasks in unstructured dynamic environments, such as navigating through complex obstacles and grasping different objects. Traditional methods for mobility and manipulation often treat perception, planning, and obstacle avoidance as separate processes, leading to decision delays, error accumulation, and a lack of full-body coordination. This paper proposes a new framework that trains a single model through reinforcement learning, allowing the robot to control both its body and arm movements simultaneously and actively choose when and what to perceive, thereby achieving efficient navigation and operation in complex environments.

The main contributions of the paper include:
1. **Proposing a new mobile manipulation framework**: This framework uses an active vision system, enabling the robot to consciously perceive and respond to the environment.
2. **Achieving full-body coordination**: The robot can utilize its mobility and visual capabilities not only to navigate around obstacles but also to choose "when" to perceive "what," thereby achieving efficient navigation and operation.
3. **Demonstrating performance in complex environments**: Experimental results show that this method outperforms traditional methods in both simulated and real-world environments, particularly excelling in handling dynamic obstacles.

Through these contributions, the paper aims to address the challenges of mobile manipulation systems in unstructured and dynamic environments, advancing the development of robotics technology in practical applications.