Abstract:To achieve human-level dexterity, robots must infer spatial awareness from multimodal sensing to reason over contact interactions. During in-hand manipulation of novel objects, such spatial awareness involves estimating the object's pose and shape. The status quo for in-hand perception primarily employs vision, and restricts to tracking a priori known objects. Moreover, visual occlusion of objects in-hand is imminent during manipulation, preventing current systems to push beyond tasks without occlusion. We combine vision and touch sensing on a multi-fingered hand to estimate an object's pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. We study multimodal in-hand perception in simulation and the real-world, interacting with different objects via a proprioception-driven policy. Our experiments show final reconstruction F-scores of $81$% and average pose drifts of $4.7\,\text{mm}$, further reduced to $2.3\,\text{mm}$ with known CAD models. Additionally, we observe that under heavy visual occlusion we can achieve up to $94$% improvements in tracking compared to vision-only methods. Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation. We release our evaluation dataset of 70 experiments, FeelSight, as a step towards benchmarking in this domain. Our neural representation driven by multimodal sensing can serve as a perception backbone towards advancing robot dexterity. Videos can be found on our project website <a class="link-external link-https" href="https://suddhu.github.io/neural-feels/" rel="external noopener nofollow">this https URL</a>

Action Augmentation of Tactile Perception for Soft-Body Palpation

Dexterous Manoeuvre Through Touch in a Cluttered Scene

Beyond End-Effector: Utilizing High-Resolution Tactile Signals for Physical Human–Robot Interaction

Behavioral characteristics of manual palpation to localize hard nodules in soft tissues

Haptic Glove and Platform with Gestural Control For Neuromorphic Tactile Sensory Feedback In Medical Telepresence †

Transparent Pneumatic Tactile Sensors for Soft Biomedical Robotics

Towards Robotised Palpation for Cancer Detection through Online Tissue Viscoelastic Characterisation with a Collaborative Robotic Arm

Face mediated human-robot interaction for remote medical examination

Built Different: Tactile Perception to Overcome Cross-Embodiment Capability Differences in Collaborative Manipulation

Use of tactile feedback to control exploratory movements to characterize object compliance

SeeBelow: Sub-dermal 3D Reconstruction of Tumors with Surgical Robotic Palpation and Tactile Exploration

Interpreting and Predicting Tactile Signals for the SynTouch BioTac

Punyo-1: Soft tactile-sensing upper-body robot for large object manipulation and physical human interaction

Multi-silicone bilateral soft physical twin as an alternative to traditional user interfaces for remote palpation: a comparative study

Virtual Reality Based Tactile Sensing Enhancements for Bilateral Teleoperation System with In-Hand Manipulation

Visuohaptic augmented feedback for enhancing motor skills acquisition

Cognitive Process during Palpation and Basic Concept of Remote Palpation System

Haptic Stiffness Perception Using Hand Exoskeletons in Tactile Robotic Telemanipulation

NeuralFeels with neural fields: Visuotactile perception for in-hand manipulation.

Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation

A Multifunctional Tactile Sensory System for Robotic Intelligent Identification and Manipulation Perception