Abstract:Spatial–temporal action detection in videos is a challenging problem that has attracted considerable attention in recent years. Most current approaches address action detection as an object detection problem, which utilizes successful object detection frameworks such as Faster R-CNN to operate action detection at every single frame first, and then generates action tubes by linking bounding boxes across the whole video in an offline fashion. However, unlike object detection in static images, temporal context information is vital for action detection in videos. Therefore, we propose an online action detection model that leverages the spatial–temporal context information existing in videos to perform action inference and localization. More specifically, we try to depict the spatial–temporal context pattern of actions via an encoder–decoder model that is based on a convolutional recurrent neural network. The model accepts a video snippet as input and encodes the dynamic information inside the snippet in the forward pass. During the backward pass, the decoder resolves the information for action detection with the current appearance or motion cue at each time stamp. In addition, we devise an incremental action-tube construction algorithm that enables our model to accomplish action prediction ahead of time and performs action detection in an online fashion. To evaluate the performance of our method, we conduct experiments on three popular public datasets UCF-101, UCF-Sports, and J-HMDB-21. The experimental results demonstrate that our method can achieve competitive or superior performance when compared to the state-of-the-art methods. To encourage further research, we release our project on “https://github.com.hjjpku.OATD.”

TQRFormer: Tubelet query recollection transformer for action detection

TubeR: Tubelet Transformer for Video Action Detection

Online Action Tube Detection Via Resolving The Spatio-Temporal Context Pattern

Discovering Spatio-Temporal Action Tubes

Actions as Moving Points

Detecting Action Tubes Via Spatial Action Estimation and Temporal Path Inference.

Sparse Action Tube Detection

TEDdet: Temporal Feature Exchange and Difference Network for Online Real-Time Action Detection

Finding Action Tubes with a Sparse-to-Dense Framework

ReAct: Temporal Action Detection with Relational Queries

Deformable Tube Network for Action Detection in Videos

SVFormer: Semi-supervised Video Transformer for Action Recognition

Spatial–Temporal Context-Aware Online Action Detection and Prediction

Question-Aware Tube-Switch Network for Video Question Answering

YoTube: Searching Action Proposal Via Recurrent and Static Regression Networks

Relaxed Transformer Decoders for Direct Action Proposal Generation

ActionFormer: Localizing Moments of Actions with Transformers

TSA-Net: Tube Self-Attention Network for Action Quality Assessment

OadTR: Online Action Detection with Transformers

ConvTransformer Attention Network for temporal action detection

STMixer: A One-Stage Sparse Action Detector