Gesture Recognition Using Visible Light on Mobile Devices
Zimo Liao,Zhicheng Luo,Qianyi Huang,Linfeng Zhang,Fan Wu,Qian Zhang,Guihai Chen
DOI: https://doi.org/10.1109/tnet.2024.3369996
2024-01-01
IEEE/ACM Transactions on Networking
Abstract:In-air gesture control extends a touch screen and enables contactless interaction, thus has become a popular research direction in the past few years. Prior work has implemented this functionality based on cameras, acoustic signals, and Wi-Fi via existing hardware on commercial devices. However, these methods have low user acceptance. Solutions based on cameras and acoustic signals raise privacy concerns, while WiFi-based solutions are vulnerable to background noise. As a result, these methods are not commercialized and recent flagship smartphones have implemented in-air gesture recognition by adding extra hardware on-board, such as mmWave radar and depth camera. The question is, can we support in-air gesture control on legacy devices without any hardware modifications? To answer this question, in this work, we propose, an in-air gesture recognition system leveraging the screen and ambient light sensor (ALS), which are ordinary modalities on mobile devices. For the transmitter side, we design a screen display mechanism to embed spatial information and preserve the viewing experience; for the receiver side, we develop a framework to recognize gestures from low-quality ALS readings. We implement and evaluate on both a tablet and several smartphones. Results show that can recognize $9$ types of frequently used in-air gestures with an average accuracy of $96.1\%$ .
telecommunications,computer science, theory & methods,engineering, electrical & electronic, hardware & architecture