Touch100k: A Large-Scale Touch-Language-Vision Dataset for Touch-Centric Multimodal Representation

Ning Cheng,Changhao Guan,Jing Gao,Weihao Wang,You Li,Fandong Meng,Jie Zhou,Bin Fang,Jinan Xu,Wenjuan Han

2024-06-06

Abstract:Touch holds a pivotal position in enhancing the perceptual and interactive capabilities of both humans and robots. Despite its significance, current tactile research mainly focuses on visual and tactile modalities, overlooking the language domain. Inspired by this, we construct Touch100k, a paired touch-language-vision dataset at the scale of 100k, featuring tactile sensation descriptions in multiple granularities (i.e., sentence-level natural expressions with rich semantics, including contextual and dynamic relationships, and phrase-level descriptions capturing the key features of tactile sensations). Based on the dataset, we propose a pre-training method, Touch-Language-Vision Representation Learning through Curriculum Linking (TLV-Link, for short), inspired by the concept of curriculum learning. TLV-Link aims to learn a tactile representation for the GelSight sensor and capture the relationship between tactile, language, and visual modalities. We evaluate our representation's performance across two task categories (namely, material property identification and robot grasping prediction), focusing on tactile representation and zero-shot touch understanding. The experimental evaluation showcases the effectiveness of our representation. By enabling TLV-Link to achieve substantial improvements and establish a new state-of-the-art in touch-centric multimodal representation learning, Touch100k demonstrates its value as a valuable resource for research. Project page: <a class="link-external link-https" href="https://cocacola-lab.github.io/Touch100k/" rel="external noopener nofollow">this https URL</a>.

Robotics

What problem does this paper attempt to address?

The problem this paper attempts to address is the current neglect of the language modality in haptic research and the lack of large-scale haptic-language-visual multimodal datasets. Specifically, although haptics play a crucial role in enhancing the perception and interaction capabilities of humans and robots, existing haptic research mostly focuses on visual and haptic modalities, ignoring the role of the language modality. To fill this gap, the authors have constructed a large-scale haptic-language-visual dataset named Touch100k, which contains 100,000 paired haptic, language, and visual data. This dataset aims to promote the application of haptic perception in robotics and enhance the understanding and utilization of haptic information through multimodal representation learning.

In addition, based on the Touch100k dataset, the authors propose a pre-training method—Tactile-Language-Visual Representation Learning via Curriculum Link (TLV-Link). This method aims to learn haptic representations for the GelSight sensor and capture the relationships between haptic, language, and visual modalities. The effectiveness of the proposed method is demonstrated through evaluations on two tasks: material property recognition and robotic grasp prediction. Overall, the goal of this paper is to advance haptic perception in robotics by creating a high-quality dataset and an effective pre-training method.