Abstract:Repetitive Action Counting (RAC) aims to count the number of repetitive actions occurring in videos. In the real world, repetitive actions have great diversity and bring numerous challenges (e.g., viewpoint changes, non-uniform periods, and action interruptions). Existing methods based on the temporal self-similarity matrix (TSSM) for RAC are trapped in the bottleneck of insufficient capturing action periods when applied to complicated daily videos. To tackle this issue, we propose a novel method named Hybrid Temporal Relation Modeling Network (HTRM-Net) to build diverse TSSM for RAC. The HTRM-Net mainly consists of three key components: bi-modal temporal self-similarity matrix modeling, random matrix dropping, and local temporal context modeling. Specifically, we construct temporal self-similarity matrices by bi-modal (self-attention and dual-softmax) operations, yielding diverse matrix representations from the combination of row-wise and column-wise correlations. To further enhance matrix representations, we propose incorporating a random matrix dropping module to guide channel-wise learning of the matrix explicitly. After that, we inject the local temporal context of video frames and the learned matrix into temporal correlation modeling, which can make the model robust enough to cope with error-prone situations, such as action interruption. Finally, a multi-scale matrix fusion module is designed to aggregate temporal correlations adaptively in multi-scale matrices. Extensive experiments across intra- and cross-datasets demonstrate that the proposed method not only outperforms current state-of-the-art methods but also exhibits robust capabilities in accurately counting repetitive actions in unseen action categories. Notably, our method surpasses the classical TransRAC method by 20.04\% in MAE and 22.76\% in OBO.

Context-Aware and Scale-Insensitive Temporal Repetition Counting

Rethinking temporal self-similarity for repetitive action counting

Efficient Action Counting with Dynamic Queries

Multipath 3D-Conv encoder and temporal-sequence decision for repetitive-action counting

Skim then Focus: Integrating Contextual and Fine-grained Views for Repetitive Action Counting

FCA-RAC: First Cycle Annotated Repetitive Action Counting

Repetitive Action Counting with Hybrid Temporal Relation Modeling

Energy-Based Periodicity Mining With Deep Features for Action Repetition Counting in Unconstrained Videos

Temporal Distinct Representation Learning for Action Recognition

MultiCounter: Multiple Action Agnostic Repetition Counting in Untrimmed Videos

Play and rewind: Context-aware video temporal action proposals

OVR: A Dataset for Open Vocabulary Temporal Repetition Counting in Videos

ContextDet: Temporal Action Detection with Adaptive Context Aggregation

IVAC-P2L: Leveraging Irregular Repetition Priors for Improving Video Action Counting

Frame-Recurrent Video Crowd Counting

Spatial–Temporal Context-Aware Online Action Detection and Prediction

MCMNET: Multi-Scale Context Modeling Network for Temporal Action Detection

Every Shot Counts: Using Exemplars for Repetition Counting in Videos

Intelligent Repetition Counting for Unseen Exercises: A Few-Shot Learning Approach with Sensor Signals

Repetition Estimation

Multi-scale Dynamic Network for Temporal Action Detection.