Abstract:Sliding detection can effectively enhance the stability of robot grasping operations. Methods relying solely on 2-D vision or tactile information for sliding detection often exhibit limited applicability across diverse scenarios. Furthermore, 2-D visual sensitivity to the changes in light intensity leads to inadequate performance in overexposure scenarios. In this article, we propose a novel deep neural network called point cloud sequences and tactile sequences fusion detection network (PTFD-Net), which serves the purpose of detecting the sliding state when robots grasp objects. Specifically, we utilize an improved PointRNN algorithm and 3-D convolution to extract features from the point cloud sequences and tactile sequences to construct a multimodal sequence information prediction model. In addition, we construct a dataset that integrates point cloud sequences with tactile sequences. The impact of different point cloud sequence lengths and tactile image sizes on the performance of PTFD-Net is discussed to determine the optimal parameters. The model can attain a sliding state detection accuracy of 95.49% on the dataset. Moreover, we conduct performance comparisons with baseline algorithms under overexposure scenarios, demonstrating PTFD-Net’s superiority over baseline algorithms and achieving a sliding detection accuracy of 92.71%. Finally, we conduct robustness experiments on unknown objects under different grasping poses and occlusion scenarios, resulting in sliding detection accuracies of 86.02% and 89.43%, respectively, both surpassing single-modal models. These results demonstrate the effectiveness of PTFD-Net in extracting multimodal features, enabling it to perform better robot grasping tasks in complex environments.

Vahagn: VisuAl Haptic Attention Gate Net for slip detection

Robotic Slip Detection with Visual-Tactile Fusion Based on 3D Attention Mechanism

Visuo-Tactile-Based Slip Detection Using A Multi-Scale Temporal Convolution Network

Slip Detection with Combined Tactile and Visual Information

Generalized Visual-Tactile Transformer Network for Slip Detection

Grasp Stability Assessment Through Spatio-Temporal Attention Mechanism and Adaptive Gate Fusion

Efficient Grasp Detection Network with Gaussian-Based Grasp Representation for Robotic Manipulation

Learning to Detect Slip through Tactile Estimation of the Contact Force Field and its Entropy

Slip Detection through Fusion of Visual-Tactile Data using Swin Transformer V2

Tactile-Based Slip Detection Towards Robot Grasping.

Learning to Detect Slip with Barometric Tactile Sensors and a Temporal Convolutional Neural Network

Slip Detection in Prosthetic Hand Grasping by Using the Discrete Wavelet Transform Analysis

Maintaining Grasps within Slipping Bound by Monitoring Incipient Slip

Learning To Detect Slip For Stable Grasping

Neuromorphic Event-Based Slip Detection and suppression in Robotic Grasping and Manipulation

Slip Detection: Analysis and Calibration of Univariate Tactile Signals

Gesture Classification in Electromyography Signals for Real-Time Prosthetic Hand Control Using a Convolutional Neural Network-Enhanced Channel Attention Model

Visual-tactile Fusion for Transparent Object Grasping in Complex Backgrounds

Event-Based Multimodal Spiking Neural Network with Attention Mechanism

Self-Attention Based Visual-Tactile Fusion Learning for Predicting Grasp Outcomes

PTFD-Net: A Sliding Detection Algorithm Combining Point Cloud Sequences and Tactile Sequences Information