Abstract:The popularity of wearable devices has increased the demands for the research on first-person activity recognition. However, most of the current first-person activity datasets are built based on the assumption that only the human-object interaction (HOI) activities, performed by the camera-wearer, are captured in the field of view. Since humans live in complicated scenarios, in addition to the first-person activities, it is likely that third-person activities performed by other people also appear. Analyzing and recognizing these two types of activities simultaneously occurring in a scene is important for the camera-wearer to understand the surrounding environments. To facilitate the research on concurrent first- and third-person activity recognition (CFT-AR), we first created a new activity dataset, namely PolyU concurrent first- and third-person (CFT) Daily, which exhibits distinct properties and challenges, compared with previous activity datasets. Since temporal asynchronism and appearance gap usually exist between the first- and third-person activities, it is crucial to learn robust representations from all the activity-related spatio-temporal positions. Thus, we explore both holistic scene-level and local instance-level (person-level) features to provide comprehensive and discriminative patterns for recognizing both first- and third-person activities. On the one hand, the holistic scene-level features are extracted by a 3-D convolutional neural network, which is trained to mine shared and sample-unique semantics between video pairs, via two well-designed attention-based modules and a self-knowledge distillation (SKD) strategy. On the other hand, we further leverage the extracted holistic features to guide the learning of instance-level features in a disentangled fashion, which aims to discover both spatially conspicuous patterns and temporally varied, yet critical, cues. Experimental results on the PolyU CFT Daily dataset validate that our method achieves the state-of-the-art performance.

Jointly Learning Heterogeneous Features for RGB-D Activity Recognition.

SharedSpecific SpecificSpecific Specific Feature ExtractionFeature Extraction { Joint learningX 1 X 4 X 3 X 2 { { {

Joint Deep Learning for RGB-D Action Recognition

Action Recognition with Joints-Pooled 3D Deep Convolutional Descriptors

Feature Learning for Interaction Activity Recognition in RGBD Videos

Unsupervised Learning Spatio-temporal Features for Human Activity Recognition from RGB-D Video Data

MMSS: Multi-modal Sharable and Specific Feature Learning for RGB-D Object Recognition.

RGBD-HuDaAct: A color-depth video database for human daily activity recognition

Collecting Public RGB-D Datasets for Human Daily Activity Recognition

Learning Informative Joints For Activity Recognition

3D SMoSIFT: three-dimensional sparse motion scale invariant feature transform for activity recognition from RGB-D videos

Skeleton Focused Human Activity Recognition in RGB Video

3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks

Spatial-temporal Texture Features for 3D Human Activity Recognition Using Laser-Based RGB-D Videos

Activity Recognition from RGB-D Camera with 3D Local Spatio-temporal Features

Human Interaction Recognition Fusing Multiple Features of Depth Sequences

Combing Rgb and Depth Map Features for Human Activity Recognition

3D human behavior recognition based on spatiotemporal texture features

Holistic-Guided Disentangled Learning with Cross-Video Semantics Mining for Concurrent First-Person and Third-Person Activity Recognition.

Decoupling Heterogeneous Features for Robust 3D Interacting Hand Poses Estimation

Joint Human Detection and Head Pose Estimation Via Multistream Networks for RGB-D Videos