Abstract:Transfer of learning or leveraging a pre-trained network and fine-tuning it to perform new tasks has been successfully applied in a variety of machine intelligence fields, including computer vision, natural language processing and audio/speech recognition. Drawing inspiration from neuroscience research that suggests that both visual and tactile stimuli rouse similar neural networks in the human brain, in this work, we explore the idea of transferring learning from vision to touch in the context of 3D object recognition. In particular, deep convolutional neural networks (CNN) pre-trained on visual images are adapted and evaluated for the classification of tactile data sets. To do so, we ran experiments with five different pre-trained CNN architectures and on five different datasets acquired with different technologies of tactile sensors including BathTip, Gelsight, force-sensing resistor (FSR) array, a high-resolution virtual FSR sensor, and tactile sensors on the Barrett robotic hand. The results obtained confirm the transferability of learning from vision to touch to interpret 3D models. Due to its higher resolution, tactile data from optical tactile sensors was demonstrated to achieve higher classification rates based on visual features compared to other technologies relying on pressure measurements. Further analysis of the weight updates in the convolutional layer is performed to measure the similarity between visual and tactile features for each technology of tactile sensing. Comparing the weight updates in different convolutional layers suggests that by updating a few convolutional layers of a pre-trained CNN on visual data, it can be efficiently used to classify tactile data. Accordingly, we propose a hybrid architecture performing both visual and tactile 3D object recognition with a MobileNetV2 backbone. MobileNetV2 is chosen due to its smaller size and thus its capability to be implemented on mobile devices, such that the network can classify both visual and tactile data. An accuracy of 100% for visual and 77.63% for tactile data are achieved by the proposed architecture.

End-to-End ConvNet for Tactile Recognition Using Residual Orthogonal Tiling and Pyramid Convolution Ensemble

Efficient Spatio-Temporal Tactile Object Recognition With Randomized Tiling Convolutional Networks In A Hierarchical Fusion Strategy

Masked Visual-Tactile Pre-training for Robot Manipulation

Transfer of Learning from Vision to Touch: A Hybrid Deep Convolutional Neural Network for Visuo-Tactile 3D Object Recognition

Residual Squeeze-and-Excitation Network with Multi-scale Spatial Pyramid Module for Fast Robotic Grasping Detection

Gradient adaptive sampling and multiple temporal scale 3D CNNs for tactile object recognition

Using 3D Convolutional Neural Networks for Tactile Object Recognition with Robotic Palpation

Real-Time Recurrent Tactile Recognition: Momentum Batch-Sequential Echo State Networks

Convolutional Autoencoder for Feature Extraction in Tactile Sensing

Highly efficient recognition of similar objects based on ionic robotic tactile sensors

Deep Active Cross-Modal Visuo-Tactile Transfer Learning for Robotic Object Recognition

Mining local and global spatiotemporal features for tactile object recognition

Enhancing Tactile Sensing in Robotics: Dual-Modal Force and Shape Perception with EIT-based Sensors and MM-CNN

Enhanced robotic tactile perception with spatiotemporal sensing and logical reasoning for robust object recognition

VTFEFN: An End-to-End Visual-Tactile Feature Extraction and Fusion Network

Predict Tactile Grasp Outcomes Based on Attention and Low-rank Fusion Network

Surface Recognition With a Tactile Finger Based on Automatic Features Transferred From Deep Learning

Recent progress on tactile object recognition

Surface Texture Recognition by Deep Learning‐Enhanced Tactile Sensing

Tactile Object Recognition for Humanoid Robots Using New Designed Piezoresistive Tactile Sensor and DCNN

CNN-based Methods for Object Recognition with High-Resolution Tactile Sensors