Abstract:Articulated objects are commonly found in daily life. It is essential that robots can exhibit robust perception and manipulation skills for articulated objects in real-world robotic applications. However, existing methods for articulated objects insufficiently address noise in point clouds and struggle to bridge the gap between simulation and reality, thus limiting the practical deployment in real-world scenarios. To tackle these challenges, we propose a framework towards Robust Perception and Manipulation for Articulated Objects (RPMArt), which learns to estimate the articulation parameters and manipulate the articulation part from the noisy point cloud. Our primary contribution is a Robust Articulation Network (RoArtNet) that is able to predict both joint parameters and affordable points robustly by local feature learning and point tuple voting. Moreover, we introduce an articulation-aware classification scheme to enhance its ability for sim-to-real transfer. Finally, with the estimated affordable point and articulation joint constraint, the robot can generate robust actions to manipulate articulated objects. After learning only from synthetic data, RPMArt is able to transfer zero-shot to real-world articulated objects. Experimental results confirm our approach's effectiveness, with our framework achieving state-of-the-art performance in both noise-added simulation and real-world environments. Code, data and more results can be found on the project website at <a class="link-external link-https" href="https://r-pmart.github.io" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the problem of robust perception and manipulation of articulated objects by robots in the real world. Specifically, the paper focuses on the following two main challenges: 1. **Point - cloud noise problem**: Point - cloud data in the real world usually contains noise, which may be caused by poor lighting conditions or depth camera measurement errors. Moreover, real - world articulated object datasets are scarce and difficult to obtain. Therefore, when only using synthetic data for training, how to bridge the gap between simulation and reality is a key issue. 2. **Semantic and physical requirements for articulated object manipulation**: The manipulation of articulated objects requires not only an understanding of the object's semantic structure (for example, identifying operable parts), but also compliance with physical constraints (for example, the range of motion of joints). These requirements make the manipulation task more complex. To solve these problems, the authors propose a framework named RPMArt, which can estimate articulated parameters from noisy point clouds and manipulate articulated parts. The following are the main contributions of RPMArt: - **Robust Articulated Network (RoArtNet)**: Through local feature learning and point - tuple voting, RoArtNet can robustly predict joint parameters and affordable points, thereby effectively transferring simulation results to real - world scenarios. - **Physical - guided manipulation based on operability**: By combining the estimated operable points and joint constraints, the robot can generate effective and robust actions to manipulate articulated objects. - **Comprehensive experimental verification**: Extensive experiments have been carried out in simulation and real - world environments, demonstrating the superior performance of this method in perception and manipulation tasks. ### Specific goals of the paper The specific goal of the paper is to develop a method that can accurately estimate the joint parameters of articulated objects in noisy point - cloud data and generate robust manipulation actions based on these parameters. In this way, the robot can successfully manipulate various articulated objects in the real world without a large amount of real - world data for training. ### Summary In general, this paper is dedicated to improving the robot's ability to perceive and manipulate articulated objects in the real world, especially in the presence of noise. By introducing a robust perception network and a physical - guided manipulation strategy based on operability, RPMArt can achieve zero - sample transfer and achieve excellent performance in the real world when trained only with synthetic data.

RPMArt: Towards Robust Perception and Manipulation for Articulated Objects

MARS: Multimodal Active Robotic Sensing for Articulated Characterization

GAPartManip: A Large-scale Part-centric Dataset for Material-Agnostic Articulated Object Manipulation

Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs

Sim2Real^2: Actively Building Explicit Physics Model for Precise Articulated Object Manipulation

Tac-Man: Tactile-Informed Prior-Free Manipulation of Articulated Objects

Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model

GAMMA: Generalizable Articulation Modeling and Manipulation for Articulated Objects

AKB-48: A Real-World Articulated Object Knowledge Base

Sim2Real<SUP>2</SUP>: Actively Building Explicit Physics Model for Precise Articulated Object Manipulation

Articulated Object Manipulation with Coarse-to-fine Affordance for Mitigating the Effect of Point Cloud Noise

Articulated Object Manipulation using Online Axis Estimation with SAM2-Based Tracking

Building Rearticulable Models for Arbitrary 3D Objects from 4D Point Clouds

FlowBot++: Learning Generalized Articulated Objects Manipulation via Articulation Projection

Learning Part Motion of Articulated Objects Using Spatially Continuous Neural Implicit Representations

VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects

SM$^3$: Self-Supervised Multi-task Modeling with Multi-view 2D Images for Articulated Objects

Visual Identification of Articulated Object Parts

ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collections

Part-Guided 3D RL for Sim2Real Articulated Object Manipulation

ImageManip: Image-based Robotic Manipulation with Affordance-guided Next View Selection