Digitizing Touch with an Artificial Multimodal Fingertip

Mike Lambeta,Tingfan Wu,Ali Sengul,Victoria Rose Most,Nolan Black,Kevin Sawyer,Romeo Mercado,Haozhi Qi,Alexander Sohn,Byron Taylor,Norb Tydingco,Gregg Kammerer,Dave Stroud,Jake Khatha,Kurt Jenkins,Kyle Most,Neal Stein,Ricardo Chavira,Thomas Craven-Bartle,Eric Sanchez,Yitian Ding,Jitendra Malik,Roberto Calandra

2024-11-05

Abstract:Touch is a crucial sensing modality that provides rich information about object properties and interactions with the physical environment. Humans and robots both benefit from using touch to perceive and interact with the surrounding environment (Johansson and Flanagan, 2009; Li et al., 2020; Calandra et al., 2017). However, no existing systems provide rich, multi-modal digital touch-sensing capabilities through a hemispherical compliant embodiment. Here, we describe several conceptual and technological innovations to improve the digitization of touch. These advances are embodied in an artificial finger-shaped sensor with advanced sensing capabilities. Significantly, this fingertip contains high-resolution sensors (~8.3 million taxels) that respond to omnidirectional touch, capture multi-modal signals, and use on-device artificial intelligence to process the data in real time. Evaluations show that the artificial fingertip can resolve spatial features as small as 7 um, sense normal and shear forces with a resolution of 1.01 mN and 1.27 mN, respectively, perceive vibrations up to 10 kHz, sense heat, and even sense odor. Furthermore, it embeds an on-device AI neural network accelerator that acts as a peripheral nervous system on a robot and mimics the reflex arc found in humans. These results demonstrate the possibility of digitizing touch with superhuman performance. The implications are profound, and we anticipate potential applications in robotics (industrial, medical, agricultural, and consumer-level), virtual reality and telepresence, prosthetics, and e-commerce. Toward digitizing touch at scale, we open-source a modular platform to facilitate future research on the nature of touch.

Robotics,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The problem this paper attempts to address is the development of an artificial multimodal fingertip sensor capable of achieving tactile digitization. Specifically, the goals of the paper include:

1. **Enhancing tactile perception**: By designing an artificial fingertip with a high-resolution sensor (approximately 8.3 million sensing units), it aims to achieve high-precision perception of multi-directional touch and various signals (such as force, vibration, temperature, etc.).

2. **Real-time data processing**: Integrating an artificial intelligence neural network accelerator into the device allows the sensor to process data in real-time environments, mimicking the human reflex arc and reducing reaction delays.

3. **Multimodal perception**: The sensor can perceive not only tactile information but also sound, temperature, and even smell, providing richer environmental perception capabilities.

4. **Overcoming existing technological limitations**: Current tactile sensors typically capture only single or limited modal information, whereas the proposed artificial fingertip sensor can simultaneously capture multiple modal information, achieving superhuman levels of tactile perception performance.

5. **Wide applications**: This advanced tactile perception technology is expected to have wide applications in fields such as robotics (industrial, medical, agricultural, and consumer-grade), virtual reality and telepresence, prosthetics, and e-commerce.

Through these innovations, the paper aims to bridge the gap between the digital and physical worlds, enabling artificial intelligence systems to perceive and interact more realistically.