Abstract:Egocentric videos, which record the daily activities of individuals from a first-person point of view, have attracted increasing attention during recent years because of their growing use in many popular applications, including life logging, health monitoring and virtual reality. As a fundamental problem in egocentric vision, one of the tasks of egocentric action recognition aims to recognize the actions of the camera wearers from egocentric videos. In egocentric action recognition, relation modeling is important, because the interactions between the camera wearer and the recorded persons or objects form complex relations in egocentric videos. However, only a few of existing methods model the relations between the camera wearer and the interacting persons for egocentric action recognition, and moreover they require prior knowledge or auxiliary data to localize the interacting persons. In this work, we consider modeling the relations in a weakly supervised manner, i.e., without using annotations or prior knowledge about the interacting persons or objects, for egocentric action recognition. We form a weakly supervised framework by unifying automatic interactor localization and explicit relation modeling for the purpose of automatic relation modeling. First, we learn to automatically localize the interactors, i.e., the body parts of the camera wearer and the persons or objects that the camera wearer interacts with, by learning a series of keypoints directly from video data to localize the action-relevant regions with only action labels and some constraints on these keypoints. Second, more importantly, to explicitly model the relations between the interactors, we develop an ego-relational LSTM (long short-term memory) network with several candidate connections to model the complex relations in egocentric videos, such as the temporal, interactive, and contextual relations. In particular, to reduce human efforts and manual interventions needed to construct an optimal ego-relational LSTM structure, we search for the optimal connections by employing a differentiable network architecture search mechanism, which automatically constructs the ego-relational LSTM network to explicitly model different relations for egocentric action recognition. We conduct extensive experiments on egocentric video datasets to illustrate the effectiveness of our method.

Egocentric Early Action Prediction via Adversarial Knowledge Distillation

Distilling Interaction Knowledge for Semi-Supervised Egocentric Action Recognition

Egocentric Early Action Prediction Via Multimodal Transformer-Based Dual Action Prediction

Knowledge-guided Pre-Training and Fine-Tuning: Video Representation Learning for Action Recognition

Multimodal Distillation for Egocentric Action Recognition

Egocentric Prediction of Action Target in 3D

Deep Attention Network for Egocentric Action Recognition.

Action Knowledge Transfer for Action Prediction with Partial Videos

Early Action Prediction with Generative Adversarial Networks

Egocentric Action Recognition by Automatic Relation Modeling.

Adversarial Memory Networks for Action Prediction

Progressive Teacher-Student Learning For Early Action Prediction

Generative Model-based Feature Knowledge Distillation for Action Recognition

Toward Egocentric Compositional Action Anticipation with Adaptive Semantic Debiasing

Object Aware Egocentric Online Action Detection

Rich Action-Semantic Consistent Knowledge for Early Action Prediction

Cross-modality Online Distillation for Multi-View Action Recognition

Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition

Teaching Yourself: A Self-Knowledge Distillation Approach to Action Recognition

Privileged Knowledge Distillation for Online Action Detection