Abstract:High-resolution tactile sensing can provide accurate information about local contact in contact-rich robotic tasks. However, the deployment of such tasks in unstructured environments remains under-investigated. To improve the robustness of tactile robot control in unstructured environments, we propose and study a new concept: \textit{tactile saliency} for robot touch, inspired by the human touch attention mechanism from neuroscience and the visual saliency prediction problem from computer vision. In analogy to visual saliency, this concept involves identifying key information in tactile images captured by a tactile sensor. While visual saliency datasets are commonly annotated by humans, manually labelling tactile images is challenging due to their counterintuitive patterns. To address this challenge, we propose a novel approach comprised of three interrelated networks: 1) a Contact Depth Network (ConDepNet), which generates a contact depth map to localize deformation in a real tactile image that contains target and noise features; 2) a Tactile Saliency Network (TacSalNet), which predicts a tactile saliency map to describe the target areas for an input contact depth map; 3) and a Tactile Noise Generator (TacNGen), which generates noise features to train the TacSalNet. Experimental results in contact pose estimation and edge-following in the presence of distractors showcase the accurate prediction of target features from real tactile images. Overall, our tactile saliency prediction approach gives robust sim-to-real tactile control in environments with unknown distractors. Project page: <a class="link-external link-https" href="https://sites.google.com/view/tactile-saliency/" rel="external noopener nofollow">this https URL</a>.

Visual Attention and Haptic Control: A Cross-Study.

A Wearable Vision-To-Audio Sensory Substitution Device for Blind Assistance and the Correlated Neural Substrates

Unveiling interactions of spatial-temporal information in tactile motion perception

Virtual reality interaction based on visual attention and kinesthetic information

Visual Target Interaction Modeling with Multichannel Data Fusion

A Vision-Based Tactile Sensing System for Multimodal Contact Information Perception via Neural Network

Exploring the Effect of Fingertip Aero-Haptic Feedforward Cues in Directing Eyes-Free Target Acquisition in VR

Attention for Robot Touch: Tactile Saliency Prediction for Robust Sim-to-Real Tactile Control

Human Impedance Modulation to Improve Visuo-Haptic Perception

A Visuo-Haptic Attention Training Game with Dynamic Adjustment of Difficulty.

Investigating the Effect of Sensory Concurrency on Learning Haptic Spatiotemporal Signals

See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation

Concurrent Crossmodal Feedback Assists Target-searching: Displaying Distance Information Through Visual, Auditory and Haptic Modalities

Cross-Modal Sensory Integration of Visual-Tactile Motion Information: Instrument Design and Human Psychophysics

A Solution of Human-Computer Remote Interaction with Tactile Feedback

Multi-modal Haptic Image Recognition Based on Deep Learning

Haptics-mediated Approaches for Enhancing Sustained Attention: Framework and Challenges

Psychophysics of wearable haptic/tactile perception in a multisensory context

Multimodal human attention detection for reading from facial expression, eye gaze, and mouse dynamics

Affective Haptics and Multimodal Experiments Research

Combining Vision and Tactile Sensation for Video Prediction