Abstract:Recent works on the pretraining for robot manipulation have demonstrated that representations learning from large human manipulation data can generalize well to new manipulation tasks and environments. However, these approaches mainly focus on human vision or natural language, neglecting tactile feedback. In this article, we make an attempt to explore how to pre-train a representation model for robotic manipulation using both human manipulation visual and tactile data. We develop a system for collecting visual and tactile data, featuring a cost-effective tactile glove to capture human tactile data and Hololens2 for capturing visual data. With this system, we collect a dataset of turning bottle caps. Furthermore, we introduce a novel visual-tactile fusion network and learning strategy, with one key module to tokenize 20 sparse binary tactile signals sensing touch states for the learning of tactile context and the other key module applying the attention and mask mechanism to the interaction of visual and tactile tokens for visual-tactile representation learning. We utilize our dataset to pre-train the fusion model and embed the pre-trained model into a reinforcement learning framework for downstream tasks. Experimental results demonstrate that our pre-trained model significantly aids in learning manipulation skills. Compared to methods without pre-training, our approach achieves a success rate increase of over 60%. Additionally, when compared to current visual pre-training methods, our success rate exceeds them by more than 50%.

Low Fidelity Visuo-Tactile Pretraining Improves Vision-Only Manipulation Performance

Masked Visual-Tactile Pre-training for Robot Manipulation

VITaL Pretraining: Visuo-Tactile Pretraining for Tactile and Non-Tactile Manipulation Policies

Impact of Tactile Sensor Quantities and Placements on Learning-based Dexterous Manipulation

See to Touch: Learning Tactile Dexterity through Visual Incentives

BeadSight: An Inexpensive Tactile Sensor Using Hydro-Gel Beads

Fine Manipulation Using a Tactile Skin: Learning in Simulation and Sim-to-Real Transfer

3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing

The utility of tactile force to autonomous learning of in-hand manipulation is task-dependent

Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skins

Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation

Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play

Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor

EyeSight Hand: Design of a Fully-Actuated Dexterous Robot Hand with Integrated Vision-Based Tactile Sensors and Compliant Actuation

Imagine2touch: Predictive Tactile Sensing for Robotic Manipulation using Efficient Low-Dimensional Signals

Tactile Active Inference Reinforcement Learning for Efficient Robotic Manipulation Skill Acquisition

Sensorimotor-inspired Tactile Feedback and Control Improve Consistency of Prosthesis Manipulation in the Absence of Direct Vision

DexTouch: Learning to Seek and Manipulate Objects with Tactile Dexterity

Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulation

Manipulation via Membranes: High-Resolution and Highly Deformable Tactile Sensing and Control