Highly stretchable and sensitive strain sensor for lip-reading extraction and speech recognition

Lin Cheng,Diqing Ruan,Yongwei He,Jiayao Yang,Wei Qian,Longwei Zhu,Pindie Zhu,Huaping Wu,Aiping Liu
DOI: https://doi.org/10.1039/d3tc01136d
IF: 6.4
2023-05-18
Journal of Materials Chemistry C
Abstract:Speech recognition often relies on visual stereo images and videos, but it is often limited by light intensity and opaque occlusions such as mask (the coronavirus disease 2019 (COVID-19) pandemic). Lip speech is produced by the movement of a series of muscles in the mouth. Therefore, lip-reading extraction and decoding can be achieved by attaching non-invasive sensors to capture muscle movement. Herein, we prepared a highly sensitive and stretchable CuNWs-rGO/PDMS piezoresistive strain sensor based on biomimetic pearl "brick- mortar" structure by using two-dimensional rGO and conductive stretchable One-dimensional Cu NWs as mixed "bricks" and stretchable PDMS as "mortar". The strain sensor shows a high sensitivity with a gauge factor of 26 in the stretch ability range up to 70 % and high durability over 1000 stretching-releasing cycles. Attaching this sensor to human skin can act as a health monitoring device, detecting various human motions such as the hand clenching, elbow bending, wrist pulse beating and lip-reading motion. Effective recognition of the lip reading signal was achieved by processing the lip-reading signal using a dynamic time regularization algorithm and Euclidean distance. Finally, a full system integration to real-time analyze and feedback speech information with voice broadcast and text display function based on the strain sensor was presented for accurate speech recognition, which opens up a new possibility for translating lip motion to speech or text directly by capturing lip muscle movements. This work presents a promising way to help people with vocal cord lesions or laryngeal and lingual injuries live a happy life with barrier-free communication and will enrich the diversity of lip-language translation systems in human-machine interfaces and silent speech interfaces.
materials science, multidisciplinary,physics, applied
What problem does this paper attempt to address?