Abstract:As one of the most direct and pivotal modes of human–computer interaction (HCI), the application of surface electromyography (sEMG) signals in the domain of gesture prediction has emerged as a prominent area of research. To enhance the performance of gesture prediction system based on multichannel sEMG signals, a novel gesture prediction framework is proposed that: 1) conversion of original biological signals from multichannel sEMG into 2-D time-frequency maps is achieved through the incorporation of continuous wavelet transform (CWT) and 2) for 2-D time-frequency map inputs, a Transformer-based classification network that effectively learns local and global context information is proposed, named DIFT-Net, with the goal of implementing sEMG-based gesture prediction for robot interaction. Proposed DIFT-Net employs a dual-branch interactive fusion structure based on the Swin Transformer, enabling effective acquisition of global contextual information and local details. Additionally, an attention guidance module (AGM) and an attentional interaction module (AIM) are proposed to guide network feature extraction and fusion processes in proposed DIFT-Net. The AGM module takes intermediate features from the same stage of both branches as input and guides the network to extract more localized and detailed features through convolutional attention. Meanwhile, the AIM module integrates output features from both branches to enhance the aggregation of global context information across various scales. To substantiate the efficacy of DIFT-Net, a multichannel EMG bracelet is utilized to collect and construct an sEMG signal dataset. Experimental results demonstrate that the proposed DIFT-Net attains an accuracy of 98.36% in self-built dataset and 82.64% accuracy on the public Nanapro DB1 dataset.

Slip Detection through Fusion of Visual-Tactile Data using Swin Transformer V2

Visuo-Tactile-Based Slip Detection Using A Multi-Scale Temporal Convolution Network

Robotic Slip Detection with Visual-Tactile Fusion Based on 3D Attention Mechanism

Slip Detection with Combined Tactile and Visual Information

Learning to Detect Slip through Tactile Estimation of the Contact Force Field and its Entropy

Visual-Tactile Fusion for Robotic Stable Grasping

SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection

Vahagn: VisuAl Haptic Attention Gate Net for slip detection

Slip detection for grasp stabilisation with a multi-fingered tactile robot hand

Grasp Stability Assessment Through the Fusion of Visual and Tactile Signals Using Deep Bilinear Network.

Learning To Detect Slip For Stable Grasping

Slip Detection in Prosthetic Hand Grasping by Using the Discrete Wavelet Transform Analysis

A Transformer-Based Gesture Prediction Model via sEMG Sensor for Human–Robot Interaction

A novel tactile sensor with multimodal vision and tactile units for multifunctional robot interaction

Visual-tactile Fusion for Transparent Object Grasping in Complex Backgrounds

Learning to Detect Slip with Barometric Tactile Sensors and a Temporal Convolutional Neural Network

Visual-Force-Tactile Fusion for Gentle Intricate Insertion Tasks

MPCTrans: Multi-Perspective Cue-Aware Joint Relationship Representation for 3D Hand Pose Estimation via Swin Transformer

Grasp Stability Assessment Through Attention-Guided Cross-Modality Fusion and Transfer Learning

Learned Slip-Detection-Severity Framework using Tactile Deformation Field Feedback for Robotic Manipulation

A Vision-based Robot Grasping System