Abstract:Over the past few years, automatic recognition of human interactions has drawn significant attention from researchers working in the field of Artificial Intelligence (AI). And feature extraction is one of the most critical tasks in developing efficient Human Interaction Recognition (HIR) systems. Moreover, recent researches in computer vision suggest that robust features lead to higher recognition accuracies. Hence, an improved HIR system has been proposed in this paper that combines 2D and 3D features extracted using machine learning and deep learning techniques. These discriminative features result in accurate classification and help avoid misclassification of similar interactions. Ten keyframes have been extracted from each video to reduce computational complexity. Next, these frames have been preprocessed using image normalization and noise removal techniques. The Region Of Interest (ROI), which contains the two humans involved in the interaction, has been extracted using motion detection. Then, the human silhouettes have been segmented using the GrabCut algorithm. Next, the extracted silhouettes have been converted into 3D meshes and their heat kernel signatures (HKS) have been obtained to extract key body points. A Convolutional Neural Network (CNN) has been used to extract full-body features from 2D full-body silhouettes. Then, topological and geometric features have been extracted from the key body points. Finally, the combined feature vector has been fed into Long Short-Term Memory (LSTM) and each interaction has been recognized using a Softmax classifier. The proposed system has been validated via extensive experimentation on three challenging RGB+D datasets. The recognition accuracies of 91.63%, 90.54%, and 90.13% have been achieved with the SBU Kinect Interaction, NTU RGB+D, and ISR-UoL 3D social activity datasets respectively. The results of extensive experiments performed on the proposed system suggest that it can be used effectively for various applications, such as security, surveillance, health monitoring, and assisted living.

Detection of collaborative activity with Kinect depth cameras

Characterization of a Multi-User Indoor Positioning System Based on Low Cost Depth Vision (Kinect) for Monitoring Human Activity in a Smart Home

Deep Learning Based Abnormal Behavior Detection for Elderly Healthcare Using Consumer Network Cameras

Activity Detection in Indoor Environments Using Multiple 2D Lidars

Collaborative Fall Detection Using Smart Phone and Kinect

Vision-Based Human Activity Recognition System Using Depth Silhouettes: A Smart Home System for Monitoring the Residents

Indoor 3D Video Monitoring Using Multiple Kinect Depth-Cameras

Collaborative Fall Detection Using a Wearable Device and a Companion Robot.

Detecting Abnormal Events on Binary Sensors in Smart Home Environments

Feasibility of a kinect-based system in assessing physical function of the elderly for home-based care

Resident identification using kinect depth image data and fuzzy clustering techniques

Skeletal Data Matching and Merging from Multiple RGB-D Sensors for Room-Scale Human Behaviour Tracking

Fusion of Multiple Sensors Sources in a Smart Home to Detect Scenarios of Activities in Ambient Assisted Living

Indoor Occupant Behaviour Monitoring with the Use of a Depth Registration Camera

MISO: Monitoring Inactivity of Single Older Adults at Home using RGB-D Technology

Multimodal Monitoring of Activities of Daily Living for Elderly Care

Multi-camera Egocentric Activity Detection for Personal Assistant.

Multilevel Depth and Image Fusion for Human Activity Detection.

A Kinect-based system for cognitive rehabilitation exercises monitoring

Depth-color fusion strategy for 3-D scene modeling with Kinect

An LSTM-Based Approach for Understanding Human Interactions Using Hybrid Feature Descriptors Over Depth Sensors