Abstract:Manipulating garments and fabrics has long been a critical endeavor in the development of home-assistant robots. However, due to complex dynamics and topological structures, garment manipulations pose significant challenges. Recent successes in reinforcement learning and vision-based methods offer promising avenues for learning garment manipulation. Nevertheless, these approaches are severely constrained by current benchmarks, which offer limited diversity of tasks and unrealistic simulation behavior. Therefore, we present GarmentLab, a content-rich benchmark and realistic simulation designed for deformable object and garment manipulation. Our benchmark encompasses a diverse range of garment types, robotic systems and manipulators. The abundant tasks in the benchmark further explores of the interactions between garments, deformable objects, rigid bodies, fluids, and human body. Moreover, by incorporating multiple simulation methods such as FEM and PBD, along with our proposed sim-to-real algorithms and real-world benchmark, we aim to significantly narrow the sim-to-real gap. We evaluate state-of-the-art vision methods, reinforcement learning, and imitation learning approaches on these tasks, highlighting the challenges faced by current algorithms, notably their limited generalization capabilities. Our proposed open-source environments and comprehensive analysis show promising boost to future research in garment manipulation by unlocking the full potential of these methods. We guarantee that we will open-source our code as soon as possible. You can watch the videos in supplementary files to learn more about the details of our work. Our project page is available at: <a class="link-external link-https" href="https://garmentlab.github.io/" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenges faced by current domestic assistive robots in garment manipulation. Specifically, the paper focuses on the following key issues: 1. **Infinity and complexity of garment states**: Each garment has nearly infinite states and complex dynamic characteristics. Therefore, the model needs to understand various self - deforming states of the garment, which usually requires a large amount of training data. 2. **Interaction between multiple objects**: Garment manipulation involves interaction with various types of objects (such as rigid bodies, articulated objects, fluids, and human bodies). It is of great significance to enable the model to understand the interaction between these different physical media. 3. **Sim - to - Real Gap**: Due to the visual perception complexity and diverse states of garments, garment manipulation faces a larger Sim - to - Real Gap. Existing algorithms perform poorly when migrating from the simulation environment to the real world. To address these challenges, the paper proposes **GarmentLab**, a unified simulation and benchmarking platform aimed at providing rich content and a realistic simulation environment for garment manipulation. The main contributions of GarmentLab include: - **GarmentLab environment**: A realistic and rich garment - manipulation environment that supports multiple physical simulation methods, assets, object physical properties, and multi - material interactions. - **GarmentLab benchmark**: A benchmark covering a large number of garment - manipulation tasks, providing the first real - world garment - manipulation benchmark that can be replicated internationally. - **Sim - to - Real method integration**: Integration of multiple methods to narrow the simulation - reality gap, promoting the development of real - world applications. - **Extensive experiments and detailed analysis**: Extensive experiments and detailed analysis of different types of garment - manipulation algorithms, providing valuable guidance for future research. Through these contributions, GarmentLab aims to promote research in the field of garment manipulation and improve the performance and generalization ability of existing algorithms.

GarmentLab: A Unified Simulation and Benchmark for Garment Manipulation

UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation Via Dense Visual Correspondence

ClothPose: A Real-world Benchmark for Visual Analysis of Garment Pose Via an Indirect Recording Solution.

Benchmarking the Sim-to-Real Gap in Cloth Manipulation

GarmentTracking: Category-Level Garment Pose Tracking

GARField: Addressing the visual Sim-to-Real gap in garment manipulation with mesh-attached radiance fields

Flat'n'Fold: A Diverse Multi-Modal Dataset for Garment Perception and Manipulation

Learning to Grasp Clothing Structural Regions for Garment Manipulation Tasks

RoboHanger: Learning Generalizable Robotic Hanger Insertion for Diverse Garments

Dynamic Cloth Folding Using Curriculum Learning

Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics

Garment Recognition and Reconstruction Using Object Simultaneous Localization and Mapping

Garment4D: Garment Reconstruction from Point Cloud Sequences

DiffAvatar: Simulation-Ready Garment Optimization with Differentiable Simulation

Garment3DGen: 3D Garment Stylization and Texture Generation

General-purpose Clothes Manipulation with Semantic Keypoints

GarmNet: Improving Global with Local Perception for Robotic Laundry Folding

Simpler Learning of Robotic Manipulation of Clothing by Utilizing DIY Smart Textile Technology

Household Cloth Object Set: Fostering Benchmarking in Deformable Object Manipulation

GarmentCodeData: A Dataset of 3D Made-to-Measure Garments With Sewing Patterns

Visual-Tactile Learning of Garment Unfolding for Robot-Assisted Dressing