DynaKey: Dynamic Keystroke Tracking Using a Head-Mounted Camera Device
Hao Zhang,Yafeng Yin,Lei Xie,Tao Gu,Minghui You,Sanglu Lu
DOI: https://doi.org/10.1109/jiot.2021.3114224
IF: 10.6
2022-01-01
IEEE Internet of Things Journal
Abstract:Mobile and wearable devices have become more and more popular. However, the tiny touch screen leads to inefficient interaction with these devices, especially for text input. In this article, we propose DynaKey, which allows people to type on a virtual keyboard printed on a piece of article or drawn on a desk, for inputting text into a head-mounted camera device (e.g., smart glasses). By using the built-in camera and gyroscope, we capture image frames during typing and detect possible head movements, then track keys, detect fingertips, and locate keystrokes. To track the changes of keys’ coordinates in images caused by natural head (i.e., camera) movements, we introduce perspective transformation to transform keys’ coordinates among different frames. To detect and locate keystrokes, we utilize the variation of fingertip’s coordinates across multiple frames to detect possible keystrokes for localization. To reduce the time cost, we combine gyroscope and camera to adaptively track the keys, and introduce a series of optimizations, such as keypoint detection, frame skipping, multithread processing, etc. Finally, we implement DynaKey on Android-powered devices. The extensive experimental results show that our system can efficiently track and locate the keystrokes in real time. Specifically, the average tracking deviation of the keyboard layout is less than 3 pixels and the Intersection over Union (IoU) of a key in two consecutive images is above 93%. The average keystroke localization accuracy reaches 95.5%.