Abstract:Recognition of human-object interactions is practically important in various human-centric sensing scenarios such as smart supermarket, factory, and home. This paper proposes an RF-Camera system by fusing RFID and Computer Vision (CV) techniques, which is the first work to recognize the human gestural interactions with physical objects in multi-subject and multi-object scenarios. In RF-Camera, we first propose a dimension reduction method to transform the subject's 3D hand trajectory captured by depth camera to a 2D image, using which the subject's gesture can be recognized. We also propose a method to extract the facial image of target subject from an image that may contain irrelevant subjects, thereby further recognizing his/her identity. Finally, we model the physical movements of the held object's tag and further predict the tag phase data, by comparing which with real phase data of each tag human-object matching can be discovered. When implementing RF-Camera, three technical challenges need to be addressed. (i) To remove noisy data corresponding to irrelevant actions from raw sensing data, we propose a state transition diagram to determine the boundary of effective data. (ii) To predict phase data of the held target tag with unknown hand-tag offset, we quantify target tag trajectory by adding a variable hand-tag vector to captured hand trajectory. (iii) To ensure high reading rates of target tags in tag-dense scenarios, we propose a CV-assisted RFID scheduling method, in which analytics on CV data can help schedule RFID readings. We conduct extensive experiments to evaluate the performance of RF-Camera. Experimental results demonstrate that RF-Camera can recognize the gestural actions, human identity and human-object matching with an average accuracy higher than 90% in most cases.

Gesture Recognition System Based on RFID.

Research on Key Technologies of RFID Based Device Free Gesture Recognition

Gesture Recognition with RFID: an Experimental Study.

A RD-T Network for Hand Gesture Recognition Based on Millimeter-Wave Sensor

GRfid: A Device-Free RFID-Based Gesture Recognition System

Gesture Recognition System Based on Neural Networks by Using COTS RFID Tag Array.

Towards Domain-independent Complex and Fine-grained Gesture Recognition with RFID.

Wear-free Gesture Recognition Based on Residual Features of RFID Signals

RF-CGR: Enable Chinese Character Gesture Recognition With RFID

RF-Eletter: A Cross-Domain English Letter Recognition System Based on RFID

RFnet: Automatic Gesture Recognition and Human Identification Using Time Series RFID Signals

Real-Time and Accurate Gesture Recognition with Commercial RFID Devices

Multi - Touch in the Air: Device-Free Finger Tracking and Gesture Recognition Via COTS RFID.

Wireless Hand Gesture Recognition Based on Continuous-Wave Doppler Radar Sensors

Device-Free Gesture Recognition Using Time Series Rfid Signals

RFID Based Real-Time Recognition of Ongoing Gesture with Adversarial Learning

RF-Alphabet: Cross Domain Alphabet Recognition System Based on RFID Differential Threshold Similarity Calculation Model.

Intelligent Contactless Gesture Recognition Using WLAN Physical Layer Information

SGRS: A Sequential Gesture Recognition System Using COTS RFID

RFID and Camera Fusion for Recognition of Human-Object Interactions.

Domain-Independent Gesture Recognition Using Single-Channel Time-Modulated Array