Abstract:To achieve human-level dexterity, robots must infer spatial awareness from multimodal sensing to reason over contact interactions. During in-hand manipulation of novel objects, such spatial awareness involves estimating the object's pose and shape. The status quo for in-hand perception primarily employs vision, and restricts to tracking a priori known objects. Moreover, visual occlusion of objects in-hand is imminent during manipulation, preventing current systems to push beyond tasks without occlusion. We combine vision and touch sensing on a multi-fingered hand to estimate an object's pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. We study multimodal in-hand perception in simulation and the real-world, interacting with different objects via a proprioception-driven policy. Our experiments show final reconstruction F-scores of $81$% and average pose drifts of $4.7\,\text{mm}$, further reduced to $2.3\,\text{mm}$ with known CAD models. Additionally, we observe that under heavy visual occlusion we can achieve up to $94$% improvements in tracking compared to vision-only methods. Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation. We release our evaluation dataset of 70 experiments, FeelSight, as a step towards benchmarking in this domain. Our neural representation driven by multimodal sensing can serve as a perception backbone towards advancing robot dexterity. Videos can be found on our project website <a class="link-external link-https" href="https://suddhu.github.io/neural-feels/" rel="external noopener nofollow">this https URL</a>

Visual-Force-Tactile Fusion for Gentle Intricate Insertion Tasks

Beyond End-Effector: Utilizing High-Resolution Tactile Signals for Physical Human–Robot Interaction

Dexterous Manoeuvre Through Touch in a Cluttered Scene

See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation

Visual-Tactile Fusion for Robotic Stable Grasping

Multi-Modal Fusion in Contact-Rich Precise Tasks via Hierarchical Policy Learning

Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing

Development of Flexible Tactile Sensor for the Envelop of Curved Robotic Hand Finger in Grasping Force Sensing

Multimodal tactile sensing fused with vision for dexterous robotic housekeeping

3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing

A novel tactile sensor with multimodal vision and tactile units for multifunctional robot interaction

Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor

Visuo-Tactile Exploration of Unknown Rigid 3D Curvatures by Vision-Augmented Unified Force-Impedance Control

Analysing the Interplay of Vision and Touch for Dexterous Insertion Tasks

Vision-force-fused curriculum learning for robotic contact-rich assembly tasks

A Compact Fingertip-Shaped Visuo-Tactile Sensor for Manipulator Adaptive Grasping Control

Adaptive visual–tactile fusion recognition for robotic operation of multi-material system

NeuralFeels with neural fields: Visuotactile perception for in-hand manipulation.

Development of Sensory-Motor Fusion-Based Manipulation and Grasping Control for a Robotic Hand-Eye System

A Multifunctional Tactile Sensory System for Robotic Intelligent Identification and Manipulation Perception

Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation