Abstract:Human feelings expressed through verbal (e.g. voice) and non-verbal communication channels (e.g. face or body) can influence either human actions or interactions. In the literature, most of the attention was given to facial expressions for the analysis of emotions conveyed through non-verbal behaviors. Despite this, psychology highlights that the body is an important indicator of the human affective state in performing daily life activities. Therefore, this paper presents a novel method for affective action and interaction recognition from videos, exploiting multi-view representation learning and only full-body handcrafted characteristics selected following psychological and proxemic studies. Specifically, 2D skeletal data are extracted from RGB video sequences to derive diverse low-level skeleton features, i.e. multi-views, modeled through the bag-of-visual-words clustering approach generating a condition-related codebook. In this way, each affective action and interaction within a video can be represented as a frequency histogram of codewords. During the learning phase, for each affective class, training samples are used to compute its global histogram of codewords stored in a database and later used for the recognition task. In the recognition phase, the video frequency histogram representation is matched against the database of class histograms and classified as the closest affective class in terms of Euclidean distance. The effectiveness of the proposed system is evaluated on a specifically collected dataset containing 6 emotion for both actions and interactions, on which the proposed system obtains 93.64% and 90.83% accuracy, respectively. In addition, the devised strategy also achieves in line performances with other literature works based on deep learning when tested on a public collection containing 6 emotions plus a neutral state, demonstrating the effectiveness of the presented approach and confirming the findings in psychological and proxemic studies.

A multimodal approach for human activity recognition based on skeleton and RGB data

A Human Activity Recognition System Based on Dynamic Clustering of Skeleton Data

Skeleton Focused Human Activity Recognition in RGB Video

Human motion recognition using three-dimensional skeleton model based on RGBD vision system

Multi-view Multi-modal Approach Based on 5S-CNN and BiLSTM Using Skeleton, Depth and RGB Data for Human Activity Recognition

Human Action Recognition with Contextual Constraints Using a RGB-D Sensor

Skeleton-Indexed Deep Multi-Modal Feature Learning for High Performance Human Action Recognition

Enhancing Human Activity Recognition through Integrated Multimodal Analysis: A Focus on RGB Imaging, Skeletal Tracking, and Pose Estimation

Online Robust Action Recognition Based on a Hierarchical Model

Multimodal People Re-Identification Using 3D Skeleton, Depth, and Color Information

Robust Activity Recognition Based on Human Skeleton for Video Surveillance

Recurrent Network Solutions for Human Posture Recognition Based on Kinect Skeletal Data

A Multimodal Fusion Approach for Human Activity Recognition

Human Activity Recognition by Sequences of Skeleton Features

A Novel Two-Stream Transformer-Based Framework for Multi-Modality Human Action Recognition

Human-centric multimodal fusion network for robust action recognition

Supervised Learning for Human Action Recognition from Multiple Kinects

Affective Action and Interaction Recognition by Multi-View Representation Learning from Handcrafted Low-Level Skeleton Features

Vision-Based Human Activity Recognition System Using Depth Silhouettes: A Smart Home System for Monitoring the Residents

Multi-Stream Interaction Networks for Human Action Recognition

A novel approach for simultaneous human activity recognition and pose estimation via skeleton-based leveraging WiFi CSI with YOLOv8 and mediapipe frameworks