GPT-Fabric: Smoothing and Folding Fabric by Leveraging Pre-Trained Foundation Models

Vedant Raval,Enyu Zhao,Hejia Zhang,Stefanos Nikolaidis,Daniel Seita

2024-10-27

Abstract:Fabric manipulation has applications in folding blankets, handling patient clothing, and protecting items with covers. It is challenging for robots to perform fabric manipulation since fabrics have infinite-dimensional configuration spaces, complex dynamics, and may be in folded or crumpled configurations with severe self-occlusions. Prior work on robotic fabric manipulation relies either on heavily engineered setups or learning-based approaches that create and train on robot-fabric interaction data. In this paper, we propose GPT-Fabric for the canonical tasks of fabric smoothing and folding, where GPT directly outputs an action informing a robot where to grasp and pull a fabric. We perform extensive experiments in simulation to test GPT-Fabric against prior methods for smoothing and folding. GPT-Fabric matches the state-of-the-art in fabric smoothing, and also achieves comparable performance with most prior fabric folding methods tested, even without explicitly training on a fabric-specific dataset (i.e., zero-shot manipulation). Furthermore, we apply GPT-Fabric in physical experiments over 10 smoothing and 12 folding rollouts. Our results suggest that GPT-Fabric is a promising approach for high-precision fabric manipulation tasks

Robotics

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

The paper attempts to address the challenges faced by robots in fabric manipulation tasks such as smoothing and folding. Specifically, the main issues robots encounter when handling fabrics include:

1. **Infinite-dimensional configuration space**: Fabrics have an infinite number of possible shapes, making traditional motion planning techniques difficult to apply.
2. **Complex dynamics**: The interaction between the robot and the fabric results in complex dynamic changes, further increasing the difficulty of the task.
3. **Self-occlusion problem**: Fabrics can cause severe self-occlusion when folded or wrinkled, making visual recognition and manipulation more challenging.

To address these issues, existing research methods typically rely on highly engineered setups or learning-based approaches that require a large amount of robot-fabric interaction data. However, these methods have the following drawbacks:

- **High data demand**: They require a large amount of human demonstrations or random interaction data, which can be expensive and time-consuming in practical applications.
- **Limited generalization ability**: These methods often can only handle specific types of fabric configurations and have poor generalization to different types of fabrics or goals.

Therefore, the paper proposes a new method—GPT-Fabric, which utilizes pre-trained foundational models (such as GPT) to directly output low-level operation instructions for fabric smoothing and folding. This approach aims to reduce dependence on specific fabric datasets and achieve performance comparable to existing methods without specialized training data.