Abstract:Grasp classification is pivotal for understanding human interactions with objects, with wide-ranging applications in robotics, prosthetics, and rehabilitation. This study introduces a novel methodology utilizing a multisensory data glove to capture intricate grasp dynamics, including finger posture bending angles and fingertip forces. Our dataset comprises data collected from 10 participants engaging in grasp trials with 24 objects using the YCB object set. We evaluate classification performance under three scenarios: utilizing grasp posture alone, utilizing grasp force alone, and combining both modalities. We propose Glove-Net, a hybrid CNN-BiLSTM architecture for classifying grasp patterns within our dataset, aiming to harness the unique advantages offered by both CNNs and BiLSTM networks. This model seamlessly integrates CNNs' spatial feature extraction capabilities with the temporal sequence learning strengths inherent in BiLSTM networks, effectively addressing the intricate dependencies present within our grasping data. Our study includes findings from an extensive ablation study aimed at optimizing model configurations and hyperparameters. We quantify and compare the classification accuracy across these scenarios: CNN achieved 88.09%, 69.38%, and 93.51% testing accuracies for posture-only, force-only, and combined data, respectively. LSTM exhibited accuracies of 86.02%, 70.52%, and 92.19% for the same scenarios. Notably, the hybrid CNN-BiLSTM proposed model demonstrated superior performance with accuracies of 90.83%, 73.12%, and 98.75% across the respective scenarios. Through rigorous numerical experimentation, our results underscore the significance of multimodal grasp classification and highlight the efficacy of the proposed hybrid Glove-Net architectures in leveraging multisensory data for precise grasp recognition. These insights advance understanding of human-machine interaction and hold promise for diverse real-world applications.

A Deep Learning Method Based on Triplet Network Using Self-Attention for Tactile Grasp Outcomes Prediction.

Masked Visual-Tactile Pre-training for Robot Manipulation

A Cascaded Deep Learning Framework for Real-time and Robust Grasp Planning

Predict Tactile Grasp Outcomes Based on Attention and Low-rank Fusion Network

A Self-supervised Contrastive Learning Method for Grasp Outcomes Prediction

Grasp Prediction and Evaluation of Multi-Fingered Dexterous Hands Using Deep Learning.

Predict Robot Grasp Outcomes Based On Multi-Modal Information

Grasp Stability Prediction Using Tactile Information.

Deep Learning Method for Grasping Novel Objects Using Dexterous Hands

Robotic grasping using visual and tactile sensing

Triplet Attention Transformer for Spatiotemporal Predictive Learning

Real-time Pixel-Wise Grasp Affordance Prediction Based on Multi-Scale Context Information Fusion

Integrating High-Resolution Tactile Sensing into Grasp Stability Prediction

Depth-aware gaze-following via auxiliary networks for robotics

Digital Twin-Enabled Grasp Outcomes Assessment for Unknown Objects Using Visual-Tactile Fusion Perception

Grasp Stability Assessment Through the Fusion of Visual and Tactile Signals Using Deep Bilinear Network.

Fully Convolutional Grasp Detection Network with Oriented Anchor Box

AAGDN: Attention-Augmented Grasp Detection Network Based on Coordinate Attention and Effective Feature Fusion Method

Glove-Net: Enhancing Grasp Classification with Multisensory Data and Deep Learning Approach

Learning Accurate and Efficient Three-Finger Grasp Generation in Clutters with an Auto-Annotated Large-Scale Dataset

Tactile Grasp Stability Classification Based on Graph Convolutional Networks