Abstract:Touch is a crucial sensing modality that provides rich information about object properties and interactions with the physical environment. Humans and robots both benefit from using touch to perceive and interact with the surrounding environment (Johansson and Flanagan, 2009; Li et al., 2020; Calandra et al., 2017). However, no existing systems provide rich, multi-modal digital touch-sensing capabilities through a hemispherical compliant embodiment. Here, we describe several conceptual and technological innovations to improve the digitization of touch. These advances are embodied in an artificial finger-shaped sensor with advanced sensing capabilities. Significantly, this fingertip contains high-resolution sensors (~8.3 million taxels) that respond to omnidirectional touch, capture multi-modal signals, and use on-device artificial intelligence to process the data in real time. Evaluations show that the artificial fingertip can resolve spatial features as small as 7 um, sense normal and shear forces with a resolution of 1.01 mN and 1.27 mN, respectively, perceive vibrations up to 10 kHz, sense heat, and even sense odor. Furthermore, it embeds an on-device AI neural network accelerator that acts as a peripheral nervous system on a robot and mimics the reflex arc found in humans. These results demonstrate the possibility of digitizing touch with superhuman performance. The implications are profound, and we anticipate potential applications in robotics (industrial, medical, agricultural, and consumer-level), virtual reality and telepresence, prosthetics, and e-commerce. Toward digitizing touch at scale, we open-source a modular platform to facilitate future research on the nature of touch.

Single Fingertip Detection Using Simple Geometric Properties of the Hand Image: A Case Study for Augmented Reality in an Educational App

A Multitouch Drawing Application With Occlusion-Free Interaction Strategies

Fingertip Detection for Human Computer Interaction

Fingertip and card recognition mobile app for interactive play.

Real-time Fingertip Detection Based on Depth Data

Real-time virtual mouse system using RGB-D images and fingertip detection

FINGERTIP DETECTION FOR HUMAN-COMPUTER INTERACTION

Real Time Fingertip Detection with Kinect Depth Image Sequences.

Review on Vision Based Real Time Fingertip Detection Approaches

Tracking of Fingertips and Centres of Palm using KINECT

A touchable virtual screen interaction system with handheld Kinect camera

Fingertip Detection Method in Hand Gesture Recognition

Attention Based Dual Branches Fingertip Detection Network and Virtual Key System

Dexterous object manipulation by a multi-fingered robotic hand with visual-tactile fingertip sensors

Depth Aware Finger Tapping On Virtual Displays

Robust Fingertip Detection in a Complex Environment.

Digitizing Touch with an Artificial Multimodal Fingertip

OneTip: A Soft Tactile Interface for 6-D Fingertip Pose Acquisition in Human-Computer Interaction

TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric Vision

Real-time Fingertip Tracking and Gesture Recognition Using RGB-D Camera

Development of a Visual-Tactile Fingertip Sensor and an Object Manipulation Method using a Multi-Fingered Robotic Hand