Abstract:The popularity of wearable devices has increased the demands for the research on first-person activity recognition. However, most of the current first-person activity datasets are built based on the assumption that only the human-object interaction (HOI) activities, performed by the camera-wearer, are captured in the field of view. Since humans live in complicated scenarios, in addition to the first-person activities, it is likely that third-person activities performed by other people also appear. Analyzing and recognizing these two types of activities simultaneously occurring in a scene is important for the camera-wearer to understand the surrounding environments. To facilitate the research on concurrent first- and third-person activity recognition (CFT-AR), we first created a new activity dataset, namely PolyU concurrent first- and third-person (CFT) Daily, which exhibits distinct properties and challenges, compared with previous activity datasets. Since temporal asynchronism and appearance gap usually exist between the first- and third-person activities, it is crucial to learn robust representations from all the activity-related spatio-temporal positions. Thus, we explore both holistic scene-level and local instance-level (person-level) features to provide comprehensive and discriminative patterns for recognizing both first- and third-person activities. On the one hand, the holistic scene-level features are extracted by a 3-D convolutional neural network, which is trained to mine shared and sample-unique semantics between video pairs, via two well-designed attention-based modules and a self-knowledge distillation (SKD) strategy. On the other hand, we further leverage the extracted holistic features to guide the learning of instance-level features in a disentangled fashion, which aims to discover both spatially conspicuous patterns and temporally varied, yet critical, cues. Experimental results on the PolyU CFT Daily dataset validate that our method achieves the state-of-the-art performance.

Human Activities Prediction by Learning Combinatorial Sparse Representations.

Recognize Human Activities From Multi-Part Missing Videos

Human Activity Recognition based on Dynamic Spatio-Temporal Relations

Semantic Sequence Analysis for Human Activity Prediction.

Detecting Group Activities with Multi-Camera Context

Activity-Based Person Identification Using Discriminative Sparse Projections And Orthogonal Ensemble Metric Learning

A human activity recognition framework in videos using segmented human subject focus

A Hierarchical Spatio-Temporal Model for Human Activity Recognition.

Graph-based Sparse Coding and Embedding for Activity-Based Human Identification

Combining Sparse And Dense Descriptors With Temporal Semantic Structures For Robust Human Action Recognition

P-Laplacian Regularized Sparse Coding for Human Activity Recognition

Predicting Human Activities Using Stochastic Grammar.

Salient Pairwise Spatio-Temporal Interest Points for Real-Time Activity Recognition.

Local Spatio-Temporal Feature Based Voting Framework for Complex Human Activity Detection and Localization

Simultaneous Learning from Human Pose and Object Cues for Real-Time Activity Recognition

Holistic-Guided Disentangled Learning with Cross-Video Semantics Mining for Concurrent First-Person and Third-Person Activity Recognition.

Spatiotemporal Multi-Task Network for Human Activity Understanding.

Human activity prediction by mapping grouplets to recurrent Self-Organizing Map.

Probabilistic Semantic Retrieval for Surveillance Videos with Activity Graphs

3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks

Enhancing early action prediction in videos through temporal composition of sub-actions