Learning Dense Visual Object Descriptors to Fold Two-Dimensional Deformable Fabrics

Yang Cao,Daoxiong Gong,Jianjun Yu

DOI: https://doi.org/10.1109/CYBER59472.2023.10256537

2023-01-01

Abstract:Manipulating two-dimensional fabrics is a significant research field in recent years. Fabric manipulation presents a formidable challenge due to the intricate dynamics and high-dimensional state space inherent in the process, prior research has predominantly relied on robot learning of task-specific strategies as a means to effectively accomplish the corresponding fabric manipulation tasks. In this work, we utilize dense visual object descriptors trained on synthetic RGB images to learn visual representations for two-dimensional fabrics. Based on the learned descriptors, the robot can learn the correspondences of similar fabrics in different configurations. We apply a novel Siamese network architecture to improve the quality of learned descriptors for three types of fabrics, including square fabrics, T-shirts and shorts. By utilizing the learned descriptors, the equivalent actions in an unknown configuration can be computed based on a fabric folding demonstration in an initial configuration. We perform a series of fabric folding tasks in different colors, sizes and shapes. The policy can achieve 87.7% average task success rate across 7 different folding tasks.