Enhancing Temporal Action Localization: Advanced S6 Modeling with Recurrent Mechanism

Sangyoun Lee,Juho Jung,Changdae Oh,Sunghee Yun

2024-07-18

Abstract:Temporal Action Localization (TAL) is a critical task in video analysis, identifying precise start and end times of actions. Existing methods like CNNs, RNNs, GCNs, and Transformers have limitations in capturing long-range dependencies and temporal causality. To address these challenges, we propose a novel TAL architecture leveraging the Selective State Space Model (S6). Our approach integrates the Feature Aggregated Bi-S6 block, Dual Bi-S6 structure, and a recurrent mechanism to enhance temporal and channel-wise dependency modeling without increasing parameter complexity. Extensive experiments on benchmark datasets demonstrate state-of-the-art results with mAP scores of 74.2% on THUMOS-14, 42.9% on ActivityNet, 29.6% on FineAction, and 45.8% on HACS. Ablation studies validate our method's effectiveness, showing that the Dual structure in the Stem module and the recurrent mechanism outperform traditional approaches. Our findings demonstrate the potential of S6-based models in TAL tasks, paving the way for future research.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

### The Problem Addressed by the Paper

This paper aims to address the issue of Temporal Action Localization (TAL) in video analysis. Specifically, it focuses on accurately identifying the start and end times of actions in videos. Existing methods such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Graph Convolutional Networks (GCN), and Transformers each have their advantages but face limitations in capturing long-range dependencies and temporal causal relationships.

To overcome these challenges, the authors propose a new TAL architecture utilizing the Selective State Space Model (S6). This method integrates Feature Aggregated Bi-S6 blocks, Dual Bi-S6 structures, and a recurrent mechanism to enhance the modeling of temporal and channel dependencies without increasing parameter complexity. Experimental results show that this method achieves state-of-the-art performance on multiple benchmark datasets, including THUMOS-14, ActivityNet, FineAction, and HACS. Ablation studies indicate that the effectiveness of this method is mainly due to the dual structure and recurrent mechanism. Additionally, the research demonstrates the potential of the S6 model in TAL tasks, laying the groundwork for future studies.