Flatten: Video Action Recognition is an Image Classification task

Junlin Chen,Chengcheng Xu,Yangfan Xu,Jian Yang,Jun Li,Zhiping Shi

2024-08-17

Abstract:In recent years, video action recognition, as a fundamental task in the field of video understanding, has been deeply explored by numerous researchers.Most traditional video action recognition methods typically involve converting videos into three-dimensional data that encapsulates both spatial and temporal information, subsequently leveraging prevalent image understanding models to model and analyze these data. However,these methods have significant drawbacks. Firstly, when delving into video action recognition tasks, image understanding models often need to be adapted accordingly in terms of model architecture and preprocessing for these spatiotemporal tasks; Secondly, dealing with high-dimensional data often poses greater challenges and incurs higher time costs compared to its lower-dimensional <a class="link-external link-http" href="http://counterparts.To" rel="external noopener nofollow">this http URL</a> bridge the gap between image-understanding and video-understanding tasks while simplifying the complexity of video comprehension, we introduce a novel video representation architecture, Flatten, which serves as a plug-and-play module that can be seamlessly integrated into any image-understanding network for efficient and effective 3D temporal data modeling.Specifically, by applying specific flattening operations (e.g., row-major transform), 3D spatiotemporal data is transformed into 2D spatial information, and then ordinary image understanding models are used to capture temporal dynamic and spatial semantic information, which in turn accomplishes effective and efficient video action recognition. Extensive experiments on commonly used datasets (Kinetics-400, Something-Something v2, and HMDB-51) and three classical image classification models (Uniformer, SwinV2, and ResNet), have demonstrated that embedding Flatten provides a significant performance improvements over original model.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

This paper attempts to solve several key problems in video action recognition tasks:

1. **Model Adaptability Problem**: Traditional video action recognition methods usually need to convert videos into three - dimensional data containing spatio - temporal information, and then use popular image understanding models for modeling and analysis. However, when dealing with video action recognition tasks, these methods often need to make corresponding adjustments to the model architecture and pre - processing, which increases the workload and separates image tasks from video tasks.

2. **High - Dimensional Data Processing Problem**: Processing high - dimensional data (such as three - dimensional spatio - temporal data) is more challenging than processing low - dimensional data (such as two - dimensional images) and will bring higher time costs. Therefore, simplifying the complexity of video understanding tasks is an important requirement.

3. **Effective Modeling of Spatio - Temporal Information**: The core of video action recognition lies in how to effectively capture and understand spatio - temporal information from videos. Although existing methods have their own characteristics, there is still room for improvement in terms of efficiency and effectiveness.

To solve these problems, the author proposes a novel video representation architecture named Flatten. Flatten converts three - dimensional spatio - temporal data into two - dimensional spatial information through specific flattening operations (such as row - first transformation), so that ordinary image understanding models can capture temporal dynamics and spatial semantic information, achieving efficient and effective video action recognition.

Specifically, the main contributions of Flatten include:
- Proposing the Flatten method, which is a parameter - free plug - in transformation that can transform spatio - temporal modeling into spatial modeling, blurring the boundaries between image understanding and video understanding tasks.
- Constructing three different Flatten instances and verifying through comparative experiments that image understanding models can achieve spatio - temporal information modeling by learning data mapping relationships.
- Conducting extensive experiments on classic CNN, Transformer and hybrid CNN - Transformer models, and the results show that the performance is significantly improved after introducing the Flatten transformation. In particular, "Uniformer(2D)-S + Flatten" achieves the best recognition performance of 81.1% on the Kinetics400 dataset.

Through these improvements, the Flatten method provides a novel and efficient solution to the challenges in video action recognition.