Abstract:Touch holds a pivotal position in enhancing the perceptual and interactive capabilities of both humans and robots. Despite its significance, current tactile research mainly focuses on visual and tactile modalities, overlooking the language domain. Inspired by this, we construct Touch100k, a paired touch-language-vision dataset at the scale of 100k, featuring tactile sensation descriptions in multiple granularities (i.e., sentence-level natural expressions with rich semantics, including contextual and dynamic relationships, and phrase-level descriptions capturing the key features of tactile sensations). Based on the dataset, we propose a pre-training method, Touch-Language-Vision Representation Learning through Curriculum Linking (TLV-Link, for short), inspired by the concept of curriculum learning. TLV-Link aims to learn a tactile representation for the GelSight sensor and capture the relationship between tactile, language, and visual modalities. We evaluate our representation's performance across two task categories (namely, material property identification and robot grasping prediction), focusing on tactile representation and zero-shot touch understanding. The experimental evaluation showcases the effectiveness of our representation. By enabling TLV-Link to achieve substantial improvements and establish a new state-of-the-art in touch-centric multimodal representation learning, Touch100k demonstrates its value as a valuable resource for research. Project page: <a class="link-external link-https" href="https://cocacola-lab.github.io/Touch100k/" rel="external noopener nofollow">this https URL</a>.

Multi-label Tactile Property Analysis.

Tactile Adjective Understanding Using Structured Output-Associated Dictionary Learning

Structured Output-Associated Dictionary Learning for Haptic Understanding

Multi-View Analysis Dictionary Learning for Image Classification.

Seeing By Touching: Cross-Modal Matching For Tactile And Vision Measurements

Active Visual-Tactile Cross-Modal Matching.

Tactile Sequence Classification Using Joint Kernel Sparse Coding

Tactile Material Identification Using Semantics-Regularized Dictionary Learning

Material Identification Using Tactile Perception: A Semantics-Regularized Dictionary Learning Method

TALON: Improving Large Language Model Cognition with Tactility-Vision Fusion

Tactile Object Recognition Using Supervised Dictionary Learning

Tactile Sequence Based Object Categorization: A Bag Of Features Modeled By Linear Dynamic System With Symmetric Transition Matrix

LDS-FCM: A Linear Dynamical System Based Fuzzy C-Means Method for Tactile Recognition

Enhancing Label Correlation Feedback in Multi-Label Text Classification via Multi-Task Learning

Object Recognition Using Tactile Measurements: Kernel Sparse Coding Methods

Touch100k: A Large-Scale Touch-Language-Vision Dataset for Touch-Centric Multimodal Representation

Multi-Label Learning By Exploiting Label Correlations For Tcm Diagnosing Parkinson'S Disease

Towards Comprehensive Multimodal Perception: Introducing the Touch-Language-Vision Dataset

Extreme Kernel Sparse Learning for Tactile Object Recognition

TextToucher: Fine-Grained Text-to-Touch Generation

Visual–tactile fusion material identification using dictionary learning