Abstract:Touch is a crucial sensing modality that provides rich information about object properties and interactions with the physical environment. Humans and robots both benefit from using touch to perceive and interact with the surrounding environment (Johansson and Flanagan, 2009; Li et al., 2020; Calandra et al., 2017). However, no existing systems provide rich, multi-modal digital touch-sensing capabilities through a hemispherical compliant embodiment. Here, we describe several conceptual and technological innovations to improve the digitization of touch. These advances are embodied in an artificial finger-shaped sensor with advanced sensing capabilities. Significantly, this fingertip contains high-resolution sensors (~8.3 million taxels) that respond to omnidirectional touch, capture multi-modal signals, and use on-device artificial intelligence to process the data in real time. Evaluations show that the artificial fingertip can resolve spatial features as small as 7 um, sense normal and shear forces with a resolution of 1.01 mN and 1.27 mN, respectively, perceive vibrations up to 10 kHz, sense heat, and even sense odor. Furthermore, it embeds an on-device AI neural network accelerator that acts as a peripheral nervous system on a robot and mimics the reflex arc found in humans. These results demonstrate the possibility of digitizing touch with superhuman performance. The implications are profound, and we anticipate potential applications in robotics (industrial, medical, agricultural, and consumer-level), virtual reality and telepresence, prosthetics, and e-commerce. Toward digitizing touch at scale, we open-source a modular platform to facilitate future research on the nature of touch.

3D Multimodal Sensing and Feedback Finger Case for Immersive Dual‐Way Interaction

Multi-modal Sensing-Based Interactive Glove System for Teleoperation and VR/AR

Soft Modular Glove with Multimodal Sensing and Augmented Haptic Feedback Enabled by Materials' Multifunctionalities.

A Skin-Integrated Multimodal Haptic Interface for Immersive Tactile Feedback

Augmented tactile-perception and haptic-feedback rings as human-machine interfaces aiming for immersive interactions

Large‐Area Hand‐Covering Elastomeric Electronic Skin Sensor with Distributed Multifunctional Sensing Capability

Minimalist Design for Multi-Dimensional Pressure-Sensing and Feedback Glove with Variable Perception Communication

A Liquid Metal Based Multimodal Sensor and Haptic Feedback Device for Thermal and Tactile Sensation Generation in Virtual Reality

A Smart Artificial Finger with Multisensations of Matter, Temperature, and Proximity

Multifunctional Tactile Feedbacks Towards Compliant Robot Manipulations via 3D-Shaped Electronic Skin

Zero-Biased Bionic Fingertip E-Skin with Multimodal Tactile Perception and Artificial Intelligence for Augmented Touch Awareness

Three Degree-of-Freedom Soft Continuum Kinesthetic Haptic Display for Telemanipulation Via Sensory Substitution at the Finger

Flexible integrated sensor with asymmetric structure for simultaneously 3D tactile and thermal sensing

Haptic-feedback smart glove as a creative human-machine interface (HMI) for virtual/augmented reality applications

Digitizing Touch with an Artificial Multimodal Fingertip

Tactile Sensing and Rendering Patch with Dynamic and Static Sensing and Haptic Feedback for Immersive Communication

Sensory-glove-based human machine interface for augmented reality (ar) applications

An artificial remote tactile device with 3D depth-of-field sensation

Fiber-optic-based human-computer interaction device for VR systems

Dual-modal Tactile E-skin: Enabling Bidirectional Human-Robot Interaction via Integrated Tactile Perception and Feedback

HumanFT: A Human-like Fingertip Multimodal Visuo-Tactile Sensor