Abstract:Humans can instinctively predict whether a given grasp will be successful through visual and rich haptic feedback. Towards the next generation of smart robotic manufacturing, robots must be equipped with similar capabilities to cope with grasping unknown objects in unstructured environments. However, most existing data-driven methods take global visual images and tactile readings from the real-world system as input, making them incapable of predicting the grasp outcomes for cluttered objects or generating large-scale datasets. First, this paper proposes a visual-tactile fusion method to predict the results of grasping cluttered objects, which is the most common scenario for grasping applications. Concretely, the multimodal fusion network (MMFN) uses the local point cloud within the gripper as the visual signal input, while the tactile signal input is the images provided by two high-resolution tactile sensors. Second, collecting data in the real world is high-cost and time-consuming. Therefore, this paper proposes a digital twin-enabled robotic grasping system to collect large-scale multimodal datasets and investigates how to apply domain randomization and domain adaptation to bridge the sim-to-real transfer gap. Finally, extensive validation experiments are conducted in physical and virtual environments. The experimental results demonstrate the effectiveness of the proposed method in assessing grasp stability for cluttered objects and performing zero-shot sim-to-real policy transfer on the real robot with the aid of the proposed migration strategy.

Grasp Stability Assessment Through the Fusion of Visual and Tactile Signals Using Deep Bilinear Network.

Visual-Tactile Fusion for Robotic Stable Grasping

Grasp Stability Assessment Through Spatio-Temporal Attention Mechanism and Adaptive Gate Fusion

Grasp Stability Assessment Through Attention-Guided Cross-Modality Fusion and Transfer Learning

Tactile Grasp Stability Classification Based on Graph Convolutional Networks

Robotic grasping using visual and tactile sensing

Predict Robot Grasp Outcomes Based On Multi-Modal Information

Grasp Stability Prediction Using Tactile Information.

Grasp State Assessment of Deformable Objects Using Visual-Tactile Fusion Perception

Digital Twin-Enabled Grasp Outcomes Assessment for Unknown Objects Using Visual-Tactile Fusion Perception

VTG: A Visual-Tactile Dataset for Three-Finger Grasp

Self-Attention Based Visual-Tactile Fusion Learning for Predicting Grasp Outcomes

Integrating High-Resolution Tactile Sensing into Grasp Stability Prediction

Bilateral Cross-Modal Fusion Network for Robot Grasp Detection

Robot Grasping Stability Prediction Network based on Feature-fusion and Feature-reconstruction of Tactile Information

A hybrid deep architecture for robotic grasp detection.

Detecting Transitions from Stability to Instability in Robotic Grasping Based on Tactile Perception

Bayesian Grasp: Robotic visual stable grasp based on prior tactile knowledge

Efficient Grasp Detection Network with Gaussian-Based Grasp Representation for Robotic Manipulation

Visuo-Tactile-Based Slip Detection Using A Multi-Scale Temporal Convolution Network

A Vision-Based Tactile Sensing System for Multimodal Contact Information Perception via Neural Network