Abstract:Recognition of human-object interactions is practically important in various human-centric sensing scenarios such as smart supermarket, factory, and home. This paper proposes an RF-Camera system by fusing RFID and Computer Vision (CV) techniques, which is the first work to recognize the human gestural interactions with physical objects in multi-subject and multi-object scenarios. In RF-Camera, we first propose a dimension reduction method to transform the subject's 3D hand trajectory captured by depth camera to a 2D image, using which the subject's gesture can be recognized. We also propose a method to extract the facial image of target subject from an image that may contain irrelevant subjects, thereby further recognizing his/her identity. Finally, we model the physical movements of the held object's tag and further predict the tag phase data, by comparing which with real phase data of each tag human-object matching can be discovered. When implementing RF-Camera, three technical challenges need to be addressed. (i) To remove noisy data corresponding to irrelevant actions from raw sensing data, we propose a state transition diagram to determine the boundary of effective data. (ii) To predict phase data of the held target tag with unknown hand-tag offset, we quantify target tag trajectory by adding a variable hand-tag vector to captured hand trajectory. (iii) To ensure high reading rates of target tags in tag-dense scenarios, we propose a CV-assisted RFID scheduling method, in which analytics on CV data can help schedule RFID readings. We conduct extensive experiments to evaluate the performance of RF-Camera. Experimental results demonstrate that RF-Camera can recognize the gestural actions, human identity and human-object matching with an average accuracy higher than 90% in most cases.

RF-CGR: Enable Chinese Character Gesture Recognition With RFID

Towards Domain-independent Complex and Fine-grained Gesture Recognition with RFID.

GRfid: A Device-Free RFID-Based Gesture Recognition System

RF-Eletter: A Cross-Domain English Letter Recognition System Based on RFID

Gesture Recognition with RFID: an Experimental Study.

Gesture Recognition System Based on RFID.

A RD-T Network for Hand Gesture Recognition Based on Millimeter-Wave Sensor

Research on Key Technologies of RFID Based Device Free Gesture Recognition

Hand gesture recognition scheme based on millimeter-wave radar with convolutional neural network

RF-Alphabet: Cross Domain Alphabet Recognition System Based on RFID Differential Threshold Similarity Calculation Model.

Real-Time and Accurate Gesture Recognition with Commercial RFID Devices

SGRS: A Sequential Gesture Recognition System Using COTS RFID

Multi - Touch in the Air: Device-Free Finger Tracking and Gesture Recognition Via COTS RFID.

Wear-free Gesture Recognition Based on Residual Features of RFID Signals

XGest: Enabling Cross-Label Gesture Recognition with RF Signals.

RFnet: Automatic Gesture Recognition and Human Identification Using Time Series RFID Signals

Gesture Recognition System Based on Neural Networks by Using COTS RFID Tag Array.

Device-Free Gesture Recognition Using Time Series Rfid Signals

RF-CSign: A Chinese Sign Language Recognition System Based on Large Kernel Convolution and Normalization-Based Attention

ReActor: Real-time and Accurate Contactless Gesture Recognition with RFID

RFID and Camera Fusion for Recognition of Human-Object Interactions.