Opening Cabinets and Drawers in the Real World using a Commodity Mobile Manipulator

Arjun Gupta,Michelle Zhang,Rishik Sathua,Saurabh Gupta

2024-02-28

Abstract:Pulling open cabinets and drawers presents many difficult technical challenges in perception (inferring articulation parameters for objects from onboard sensors), planning (producing motion plans that conform to tight task constraints), and control (making and maintaining contact while applying forces on the environment). In this work, we build an end-to-end system that enables a commodity mobile manipulator (Stretch RE2) to pull open cabinets and drawers in diverse previously unseen real world environments. We conduct 4 days of real world testing of this system spanning 31 different objects from across 13 different real world environments. Our system achieves a success rate of 61% on opening novel cabinets and drawers in unseen environments zero-shot. An analysis of the failure modes suggests that errors in perception are the most significant challenge for our system. We will open source code and models for others to replicate and build upon our system.

Robotics,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

This paper attempts to address the problem of automatically opening cabinets and drawers in diverse and previously unseen real-world environments using commercial mobile manipulators (e.g., Stretch RE2). Specifically, the paper focuses on the following technical challenges:

1. **Perception**: Inferring the joint parameters of objects (such as the position of handles, rotation axes, etc.) from onboard sensors, especially for handles that are typically small and reflective.
2. **Planning**: Generating motion plans that meet task constraints, which are determined by the joint positions of objects and the trajectory of the handles.
3. **Control**: Maintaining contact with objects and performing precise operations while applying force to the environment.

The paper addresses these issues by constructing an end-to-end system capable of successfully opening previously unseen cabinets and drawers in various real-world environments. The main contributions of the research include:

- **Perception Module**: Extending the Mask RCNN model to output joint parameters.
- **Planning Module**: Extending the SeqIK trajectory optimization framework to generate whole-body motion plans.
- **Control Strategy**: Introducing a tactile feedback-based adaptation strategy to correct errors in navigation and calibration, ensuring accurate handle grasping.

Through extensive real-world testing, the system achieved a 61% success rate on 31 different objects in 10 buildings. Additionally, the paper analyzes the failure modes of the system, finding that perception errors are the primary bottleneck, particularly in detecting objects and handles. This provides important guidance for future research, including the need for more extensive datasets and more accurate perception methods.