Abstract:Temporal action localization, which requires a machine to recognize the location as well as the category of action instances in videos, has long been researched in computer vision. The main challenge of temporal action localization lies in that videos are usually long and untrimmed with diverse action contents involved. Existing state-of-the-art action localization methods divide each video into multiple action units (i.e., proposals in two-stage methods and segments in one-stage methods) and then perform action recognition/regression on each of them individually, without explicitly exploiting their relations during learning. In this paper, we claim that the relations between action units play an important role in action localization, and a more powerful action detector should not only capture the local content of each action unit but also allow a wider field of view on the context related to it. To this end, we propose a general graph convolutional module (GCM) that can be easily plugged into existing action localization methods, including two-stage and one-stage paradigms. To be specific, we first construct a graph, where each action unit is represented as a node and their relations between two action units as an edge. Here, we use two types of relations, one for capturing the temporal connections between different action units, and the other one for characterizing their semantic relationship. Particularly for the temporal connections in two-stage methods, we further explore two different kinds of edges, one connecting the overlapping action units and the other one connecting surrounding but disjointed units. Upon the graph we built, we then apply graph convolutional networks (GCNs) to model the relations among different action units, which is able to learn more informative representations to enhance action localization. Experimental results show that our GCM consistently improves the performance of existing action localization methods, including two-stage methods (e.g., CBR [15] and R-C3D [47]) and one-stage methods (e.g., D-SSAD [22]), verifying the generality and effectiveness of our GCM. Moreover, with the aid of GCM, our approach significantly outperforms the state-of-the-art on THUMOS14 (50.9 percent versus 42.8 percent). Augmentation experiments on ActivityNet also verify the efficacy of modeling the relationships between action units. The source code and the pre-trained models are available at https://github.com/Alvin-Zeng/GCM.

Localizing Activity Groups in Videos.

Learning Visual Context for Group Activity Recognition.

Activity Group Localization By Modeling The Relations Among Participants

Robust Visual Tracking Via CAMShift and Structural Local Sparse Appearance Model

Localizing Unseen Activities in Video Via Image Query

Recognizing Human Group Activities with Localized Causalities

Unveiling group activity recognition: Leveraging Local–Global Context-Aware Graph Reasoning for enhanced actor–scene interactions

Improved Actor Relation Graph based Group Activity Recognition

Discriminative Latent Models for Recognizing Contextual Group Activities

Localizing and recognizing action unit using position information of local feature

Group Activity Recognition Based on Temporal Semantic Sub-Graph Network.

Active Spatial Positions Based Hierarchical Relation Inference for Group Activity Recognition

An Analysis of the Combination of Global and Local Models for Activity Recognition in Videos

Detector-Free Weakly Supervised Group Activity Recognition

Group Activity Recognition using Unreliable Tracked Pose

Graph Convolutional Module for Temporal Action Localization in Videos

Local Spatio-Temporal Feature Based Voting Framework for Complex Human Activity Detection and Localization

Action Co-Localization In An Untrimmed Video By Graph Neural Networks

Multi-stage Aggregated Transformer Network for Temporal Language Localization in Videos

Representing Videos As Discriminative Sub-graphs for Action Recognition*

A Heat-Map-based Algorithm for Recognizing Group Activities in Videos