Enhancing QR Code System Security by Verifying the Scanner's Gripping Hand Biometric
Ruxin Wang,Long Huang,Kaitlyn Madden,Chen Wang
DOI: https://doi.org/10.1145/3643833.3656128
2024-01-01
Abstract:Because of the great convenience and being not readable to humans, Quick Response (QR) codes are increasingly being utilized to offer a variety of security applications to mobile users, such as online payments, website logins, and private data sharing. To facilitate these security applications, QR codes usually contain sensitive information, such as bank account details, credit card numbers, and personal/organizational/device data, or they are specifically designed to work with cloud servers to provide security services. However, there is currently no existing solution to verify the identity of the smartphone user who scans a QR code from a Kiosk or another phone's screen. Verifying the scanner's identity is essential to ensure that financial transactions go to the correct recipient and that sensitive data is securely shared to its intended destination. This work aims to equip QR code providers with the ability to verify human scanners' identities, facilitating authorization and auditing. When a phone is held close to scan a QR code, we utilize the front camera of the code provider (a Kiosk or phone) to simultaneously verify the scanner's hand. Instead of requiring the scanner to present a stretched palm to obtain traditional hand geometries, we find that the geometry of an individual's hand, when it grips a phone, is also identifiable. We thus design a vision-based approach to extract gripping hand biometrics. We leverage the QR code's screen to cast light onto the scanner's gripping hand, ensuring adequate illumination even in low-light conditions. We then use a hand tracking tool, MediaPipe, to detect and localize the hand and develop a transformer-based algorithm to verify four types of gripping hand biometric features extracted from the hand image, including hand contour, skeleton, color, and surface. We further capture the subtle hand joint movements for liveness validation, because the user needs to click touchscreen buttons to start QR code scanning. Extensive experiments, including a long-term study spanning over 32 months, show that the system achieves 98.3% accuracy in verifying the user and mitigating 2D and 3D replay attacks. Compared to the widely used facial recognition, this approach addresses the recent struggles of identifying faces behind masks and the public concerns about privacy erosion.