Abstract:Transfer of learning or leveraging a pre-trained network and fine-tuning it to perform new tasks has been successfully applied in a variety of machine intelligence fields, including computer vision, natural language processing and audio/speech recognition. Drawing inspiration from neuroscience research that suggests that both visual and tactile stimuli rouse similar neural networks in the human brain, in this work, we explore the idea of transferring learning from vision to touch in the context of 3D object recognition. In particular, deep convolutional neural networks (CNN) pre-trained on visual images are adapted and evaluated for the classification of tactile data sets. To do so, we ran experiments with five different pre-trained CNN architectures and on five different datasets acquired with different technologies of tactile sensors including BathTip, Gelsight, force-sensing resistor (FSR) array, a high-resolution virtual FSR sensor, and tactile sensors on the Barrett robotic hand. The results obtained confirm the transferability of learning from vision to touch to interpret 3D models. Due to its higher resolution, tactile data from optical tactile sensors was demonstrated to achieve higher classification rates based on visual features compared to other technologies relying on pressure measurements. Further analysis of the weight updates in the convolutional layer is performed to measure the similarity between visual and tactile features for each technology of tactile sensing. Comparing the weight updates in different convolutional layers suggests that by updating a few convolutional layers of a pre-trained CNN on visual data, it can be efficiently used to classify tactile data. Accordingly, we propose a hybrid architecture performing both visual and tactile 3D object recognition with a MobileNetV2 backbone. MobileNetV2 is chosen due to its smaller size and thus its capability to be implemented on mobile devices, such that the network can classify both visual and tactile data. An accuracy of 100% for visual and 77.63% for tactile data are achieved by the proposed architecture.

Applying Transfer Learning to Recognize Clothing Patterns Using a Finger-Mounted Camera

Cloth Pattern Recognition Using Machine Learning and Neural Network

HandSight: DeCAF & Improved Fisher Vectors to Classify Clothing Color and Texture with a Finger-Mounted Camera

Continuous Perception Garment Classification Based on Optical Flow Variation.

Deforming Garment Classification with Shallow Temporal Extraction and Tree-Based Fusion

Application of Clothing Design Based on Computer Vision Technology Using Hybrid Graph Convolution Neural Network and Lotus Effect Optimization Algorithm

Shape-preserving Swapping Autoencoder for Garment Texture Transfer

Multi-Task Curriculum Transfer Deep Learning of Clothing Attributes

Clothes Grasping and Unfolding Based on RGB-D Semantic Segmentation

Clothing recognition based on deep sparse convolutional neural network

Finger Angle-Based Hand Gesture Recognition for Smart Infrastructure Using Wearable Wrist-Worn Camera

Learning the Color Space for Effective Fabric Defect Detection

A Digital Simulation and Re-editing Method for Clothing Patterns Based on Deep Learning and Somatosensory Interaction

Research on Innovative Design of Traditional Clothing Handicraft under Computer Big Data Technology

Learning Visual Clothing Style with Heterogeneous Dyadic Co-occurrences

Learning to Transfer Texture from Clothing Images to 3D Humans

FFENet: Frequency-Spatial Feature Enhancement Network for Clothing Classification

Deep Finger Texture Learning for Verifying People

Fabric defect detection based on transfer learning and improved Faster R-CNN

A Federated Approach for Fine-Grained Classification of Fashion Apparel

Transfer of Learning from Vision to Touch: A Hybrid Deep Convolutional Neural Network for Visuo-Tactile 3D Object Recognition