Abstract:To achieve human-level dexterity, robots must infer spatial awareness from multimodal sensing to reason over contact interactions. During in-hand manipulation of novel objects, such spatial awareness involves estimating the object's pose and shape. The status quo for in-hand perception primarily employs vision, and restricts to tracking a priori known objects. Moreover, visual occlusion of objects in-hand is imminent during manipulation, preventing current systems to push beyond tasks without occlusion. We combine vision and touch sensing on a multi-fingered hand to estimate an object's pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. We study multimodal in-hand perception in simulation and the real-world, interacting with different objects via a proprioception-driven policy. Our experiments show final reconstruction F-scores of $81$% and average pose drifts of $4.7\,\text{mm}$, further reduced to $2.3\,\text{mm}$ with known CAD models. Additionally, we observe that under heavy visual occlusion we can achieve up to $94$% improvements in tracking compared to vision-only methods. Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation. We release our evaluation dataset of 70 experiments, FeelSight, as a step towards benchmarking in this domain. Our neural representation driven by multimodal sensing can serve as a perception backbone towards advancing robot dexterity. Videos can be found on our project website <a class="link-external link-https" href="https://suddhu.github.io/neural-feels/" rel="external noopener nofollow">this https URL</a>

PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images

PressureVision: Estimating Hand Pressure from a Single RGB Image

Visual Contact Pressure Estimation for Grippers in the Wild

EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision

FeelAnyForce: Estimating Contact Force Feedback from Tactile Sensation for Vision-Based Tactile Sensors

Visual Pressure Estimation and Control for Soft Robotic Grippers

Teaching Cameras to Feel: Estimating Tactile Physical Properties of Surfaces From Images

PseudoTouch: Efficiently Imaging the Surface Feel of Objects for Robotic Manipulation

Pressure Eye: In-bed Contact Pressure Estimation via Contact-less Imaging

TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric Vision

Grasping Force Estimation for Markerless Visuotactile Sensors

Real-time Pressure Measurement Using Vision-based Tactile Sensor

Posture-Informed Muscular Force Learning for Robust Hand Pressure Estimation

Dataset of Tactile Signatures of the Human Right Hand in Twenty-One Activities of Daily Living Using a High Spatial Resolution Pressure Sensor

Deep Tactile Experience: Estimating Tactile Sensor Output from Depth Sensor Data

NeuralFeels with neural fields: Visuotactile perception for in-hand manipulation.

Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation

Leveraging RGB-Pressure for Whole-body Human-to-Humanoid Motion Imitation

Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images

Anchors Based Method for Fingertips Position Estimation from a Monocular RGB Image using Deep Neural Network

Imagine2touch: Predictive Tactile Sensing for Robotic Manipulation using Efficient Low-Dimensional Signals