Abstract:Humans make extensive use of vision and touch as complementary senses, with vision providing global information about the scene and touch measuring local information during manipulation without suffering from occlusions. In this work, we propose a novel framework for learning multi-task visuo-tactile representations in a self-supervised manner. We design a mechanism which enables a robot to autonomously collect spatially aligned visual and tactile data, a key property for downstream tasks. We then train visual and tactile encoders to embed these paired sensory inputs into a shared latent space using cross-modal contrastive loss. The learned representations are evaluated without fine-tuning on 5 perception and control tasks involving deformable surfaces: tactile classification, contact localization, anomaly detection (e.g., surgical phantom tumor palpation), tactile search from a visual query (e.g., garment feature localization under occlusion), and tactile servoing along cloth edges and cables. The learned representations achieve an 80% success rate on towel feature classification, a 73% average success rate on anomaly detection in surgical materials, a 100% average success rate on vision-guided tactile search, and 87.8% average servo distance along cables and garment seams. These results suggest the flexibility of the learned representations and pose a step toward task-agnostic visuo-tactile representation learning for robot control.

Active Clothing Material Perception Using Tactile Sensing and Deep Learning

What Matters for Active Texture Recognition With Vision-Based Tactile Sensors

Learning to Singulate Layers of Cloth using Tactile Feedback

Lifelong Visual-Tactile Cross-Modal Learning for Robotic Material Perception

Open‐Environment Tactile Sensing System: Towards Simple and Efficient Material Identification

Human-Touch-Inspired Material Recognition for Robotic Tactile Sensing

Multimodal Material Classification for Robots using Spectroscopy and High Resolution Texture Imaging

Perception-enhancement Based Task Learning and Action Scheduling for Robotic Limb in CPS Environment

Surface Texture Recognition by Deep Learning‐Enhanced Tactile Sensing

Learning Self-Supervised Representations from Vision and Touch for Active Sliding Perception of Deformable Surfaces

Visual-Tactile Learning of Garment Unfolding for Robot-Assisted Dressing

Simpler Learning of Robotic Manipulation of Clothing by Utilizing DIY Smart Textile Technology

Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images

A Multifunctional Tactile Sensory System for Robotic Intelligent Identification and Manipulation Perception

Robot Vision Architecture for Autonomous Clothes Manipulation

MateRobot: Material Recognition in Wearable Robotics for People with Visual Impairments

RobotSweater: Scalable, Generalizable, and Customizable Machine-Knitted Tactile Skins for Robots

Texture Recognition Based on Perception Data from a Bionic Tactile Sensor

Dynamic Layer Detection of a Thin Silk Cloth using DenseTact Optical Tactile Sensors

Multimodal Surface Sensing Based on Hybrid Flexible Triboelectric and Piezoresistive Sensor.