Abstract:Visual-tactile fusion information plays a crucial role in robotic object classification. The fusion module in existing visual-tactile fusion models directly splices visual and tactile features at the feature layer; however, for different objects, the contributions of visual features and tactile features to classification are different. Moreover, direct concatenation may ignore features that are more beneficial for classification and will also increase computational costs and reduce model classification efficiency. To utilize object feature information more effectively and further improve the efficiency and accuracy of robotic object classification, we propose a visual-tactile fusion object classification method based on adaptive feature weighting in this article. First, a lightweight feature extraction module is used to extract the visual and tactile features of each object. Then, the two feature vectors are input into an adaptive weighted fusion module. Finally, the fused feature vector is input into the fully connected layer for classification, yielding the categories and physical attributes of the objects. In this article, extensive experiments are performed with the Penn Haptic Adjective Corpus 2 public dataset and the newly developed Visual-Haptic Adjective Corpus 52 dataset. The experimental results demonstrate that for the public dataset Penn Haptic Adjective Corpus 2, our method achieves a value of 0.9750 in terms of the area under the curve. Compared with the highest area under the curve obtained by the existing state-of-the-art methods, our method improves by 1.92%. Moreover, compared with the existing state-of-the-art methods, our method achieves the best results in training time and inference time; while for the novel Visual-Haptic Adjective Corpus 52 dataset, our method achieves values of 0.9827 and 0.9850 in terms of the area under the curve and accuracy metrics, respectively. Furthermore, the inference time reaches 1.559 s/sheet, demonstrating the effectiveness of the proposed method.

VTFEFN: An End-to-End Visual-Tactile Feature Extraction and Fusion Network

Fusion of Low-Illuminance Visible and Near-Infrared Images Based on Convolutional Neural Networks

Predict Tactile Grasp Outcomes Based on Attention and Low-rank Fusion Network

VITO-Transformer: A Visual-Tactile Fusion Network for Object Recognition

IVJDN: an End-to-End Network for Joint Infrared and Visible Image Fusion and Detection

TCCFusion: An Infrared and Visible Image Fusion Method based on Transformer and Cross Correlation

Visual-tactile fusion object classification method based on adaptive feature weighting

DTFusion: Infrared and Visible Image Fusion Based on Dense Residual PConv-ConvNeXt and Texture-Contrast Compensation

MFTCFNet: Infrared and Visible Image Fusion Network Based on Multi-Layer Feature Tightly Coupled

THFuse: An Infrared and Visible Image Fusion Network using Transformer and Hybrid Feature Extractor

MEEAFusion: Multi-Scale Edge Enhancement and Joint Attention Mechanism Based Infrared and Visible Image Fusion

RTFusion: A Multimodal Fusion Network with Significant Information Enhancement

Visual-Tactile Fusion for Robotic Stable Grasping

TSVFN: Two-Stage Visual Fusion Network for multimodal relation extraction

A Human-Like Siamese-Based Visual-Tactile Fusion Model for Object Recognition

HDCTfusion: Hybrid Dual-Branch Network Based on CNN and Transformer for Infrared and Visible Image Fusion

MFFNet: Multi-modal Feature Fusion Network for V-D-T Salient Object Detection

TDDFusion: A Target-Driven Dual Branch Network for Infrared and Visible Image Fusion

Efficient Spatio-Temporal Tactile Object Recognition With Randomized Tiling Convolutional Networks In A Hierarchical Fusion Strategy

HitFusion: Infrared and Visible Image Fusion for High-Level Vision Tasks Using Transformer

A Late Fusion Approach for Harnessing Multi-Cnn Model High-Level Features