Abstract:Egocentric videos, which record the daily activities of individuals from a first-person point of view, have attracted increasing attention during recent years because of their growing use in many popular applications, including life logging, health monitoring and virtual reality. As a fundamental problem in egocentric vision, one of the tasks of egocentric action recognition aims to recognize the actions of the camera wearers from egocentric videos. In egocentric action recognition, relation modeling is important, because the interactions between the camera wearer and the recorded persons or objects form complex relations in egocentric videos. However, only a few of existing methods model the relations between the camera wearer and the interacting persons for egocentric action recognition, and moreover they require prior knowledge or auxiliary data to localize the interacting persons. In this work, we consider modeling the relations in a weakly supervised manner, i.e., without using annotations or prior knowledge about the interacting persons or objects, for egocentric action recognition. We form a weakly supervised framework by unifying automatic interactor localization and explicit relation modeling for the purpose of automatic relation modeling. First, we learn to automatically localize the interactors, i.e., the body parts of the camera wearer and the persons or objects that the camera wearer interacts with, by learning a series of keypoints directly from video data to localize the action-relevant regions with only action labels and some constraints on these keypoints. Second, more importantly, to explicitly model the relations between the interactors, we develop an ego-relational LSTM (long short-term memory) network with several candidate connections to model the complex relations in egocentric videos, such as the temporal, interactive, and contextual relations. In particular, to reduce human efforts and manual interventions needed to construct an optimal ego-relational LSTM structure, we search for the optimal connections by employing a differentiable network architecture search mechanism, which automatically constructs the ego-relational LSTM network to explicitly model different relations for egocentric action recognition. We conduct extensive experiments on egocentric video datasets to illustrate the effectiveness of our method.

Slowfast Diversity-aware Prototype Learning for Egocentric Action Recognition.

Interactive Prototype Learning for Egocentric Action Recognition

Deep Attention Network for Egocentric Action Recognition.

Learning to Recognize Actions on Objects in Egocentric Video with Attention Dictionaries

Symbiotic Attention with Privileged Information for Egocentric Action Recognition

Temporal Distinct Representation Learning for Action Recognition

Symbiotic Attention for Egocentric Action Recognition with Object-Centric Alignment.

A dual-prototype network combining query-specific and class-specific attentive learning for few-shot action recognition

Multi-Stream Deep Neural Networks for RGB-D Egocentric Action Recognition

Matching Compound Prototypes for Few-Shot Action Recognition

Few-shot Action Recognition with Prototype-centered Attentive Learning

Egocentric Early Action Prediction via Adversarial Knowledge Distillation

Object Aware Egocentric Online Action Detection

Action Recognition In Rgb-D Egocentric Videos

Dear-Net: Learning Diversities for Skeleton-Based Early Action Recognition

Pattern4Ego: Learning Egocentric Video Representation Using Cross-video Activity Patterns

Egocentric Action Recognition by Automatic Relation Modeling.

Consistency Prototype Module and Motion Compensation for Few-Shot Action Recognition (CLIP-CP$\mathbf{M^2}$C)

AdaFPP: Adapt-Focused Bi-Propagating Prototype Learning for Panoramic Activity Recognition

A Prototype Learning Framework Using EMD: Application to Complex Scenes Analysis

Collaborative Spatio-temporal Feature Learning for Video Action Recognition