Abstract:To achieve human-level dexterity, robots must infer spatial awareness from multimodal sensing to reason over contact interactions. During in-hand manipulation of novel objects, such spatial awareness involves estimating the object's pose and shape. The status quo for in-hand perception primarily employs vision, and restricts to tracking a priori known objects. Moreover, visual occlusion of objects in-hand is imminent during manipulation, preventing current systems to push beyond tasks without occlusion. We combine vision and touch sensing on a multi-fingered hand to estimate an object's pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. We study multimodal in-hand perception in simulation and the real-world, interacting with different objects via a proprioception-driven policy. Our experiments show final reconstruction F-scores of $81$% and average pose drifts of $4.7\,\text{mm}$, further reduced to $2.3\,\text{mm}$ with known CAD models. Additionally, we observe that under heavy visual occlusion we can achieve up to $94$% improvements in tracking compared to vision-only methods. Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation. We release our evaluation dataset of 70 experiments, FeelSight, as a step towards benchmarking in this domain. Our neural representation driven by multimodal sensing can serve as a perception backbone towards advancing robot dexterity. Videos can be found on our project website <a class="link-external link-https" href="https://suddhu.github.io/neural-feels/" rel="external noopener nofollow">this https URL</a>

General In-Hand Object Rotation with Vision and Touch

AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch

Rotating without Seeing: Towards In-hand Dexterity through Touch

COMTIS: Customizable Touchless Interaction System for Large Screen Visualization

Rotating Objects via In-Hand Pivoting using Vision, Force and Touch

Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing

NeuralFeels with neural fields: Visuotactile perception for in-hand manipulation.

Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation

RoTip: A Finger-Shaped Tactile Sensor with Active Rotation

Eye Gaze-based Object Rotation for Head-mounted Displays.

See to Touch: Learning Tactile Dexterity through Visual Incentives

TacRot: A Parallel-Jaw Gripper with Rotatable Tactile Sensors for In-Hand Manipulation

PseudoTouch: Efficiently Imaging the Surface Feel of Objects for Robotic Manipulation

EyeSight Hand: Design of a Fully-Actuated Dexterous Robot Hand with Integrated Vision-Based Tactile Sensors and Compliant Actuation

Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes

Dynamic Modeling of Hand-Object Interactions via Tactile Sensing

Visual-Tactile Sensing for In-Hand Object Reconstruction

Incipient Slip-Based Rotation Measurement via Visuotactile Sensing During In-Hand Object Pivoting

OneTip: A Soft Tactile Interface for 6-D Fingertip Pose Acquisition in Human-Computer Interaction

Imagine2touch: Predictive Tactile Sensing for Robotic Manipulation using Efficient Low-Dimensional Signals

Learning Visuotactile Skills with Two Multifingered Hands