When Spatial meets Temporal in Action Recognition

Huilin Chen,Lei Wang,Yifan Chen,Tom Gedeon,Piotr Koniusz

2024-11-23

Abstract:Video action recognition has made significant strides, but challenges remain in effectively using both spatial and temporal information. While existing methods often focus on either spatial features (e.g., object appearance) or temporal dynamics (e.g., motion), they rarely address the need for a comprehensive integration of both. Capturing the rich temporal evolution of video frames, while preserving their spatial details, is crucial for improving accuracy. In this paper, we introduce the Temporal Integration and Motion Enhancement (TIME) layer, a novel preprocessing technique designed to incorporate temporal information. The TIME layer generates new video frames by rearranging the original sequence, preserving temporal order while embedding $N^2$ temporally evolving frames into a single spatial grid of size $N \times N$. This transformation creates new frames that balance both spatial and temporal information, making them compatible with existing video models. When $N=1$, the layer captures rich spatial details, similar to existing methods. As $N$ increases ($N\geq2$), temporal information becomes more prominent, while the spatial information decreases to ensure compatibility with model inputs. We demonstrate the effectiveness of the TIME layer by integrating it into popular action recognition models, such as ResNet-50, Vision Transformer, and Video Masked Autoencoders, for both RGB and depth video data. Our experiments show that the TIME layer enhances recognition accuracy, offering valuable insights for video processing tasks.

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenge of effectively utilizing spatial and temporal information in video action recognition. Existing methods usually focus on spatial features (such as object appearance) or temporal dynamics (such as motion), but rarely can comprehensively integrate the two. Capturing the rich temporal evolution in video frames while retaining their spatial details is crucial for improving recognition accuracy. To this end, the paper introduces a new pre - processing technique named "Temporal Integration and Motion Enhancement (TIME)" layer, which aims to combine temporal information and generate new video frames. These new frames embed \(N^2\) temporally evolving frames into an \(N\times N\) spatial grid while maintaining the temporal order, thus achieving a balance between spatial and temporal information and being compatible with existing video models.

### Specific problem description
1. **Limitations of existing methods**:
- Existing video action recognition methods often only focus on spatial features or temporal dynamics and fail to effectively integrate the two.
- Many models rely on sparse frame sampling to reduce memory usage, which may lead to the loss of critical motion information.
- Methods that highly rely on dense sampling sequences may encounter scalability problems, and methods that only focus on spatial details may perform poorly in capturing motion cues.

2. **The new method proposed in the paper**:
- The TIME layer is introduced, which is a pre - processing technique that generates new video frames by embedding temporal information into a spatial grid through rearranging the original frame sequence.
- The TIME layer can embed \(N^2\) temporally evolving frames into an \(N\times N\) spatial grid while maintaining the temporal order, thus achieving a balance between spatial and temporal information.
- When \(N = 1\), the TIME layer captures rich spatial details; as \(N\) increases (\(N\geq2\)), the temporal information becomes more prominent and the spatial information decreases accordingly to ensure compatibility with the model input.

3. **Advantages of the method**:
- The TIME layer can operate independently of the model architecture and is applicable to multiple models, including CNN, Transformer, and self - supervised learning frameworks.
- It provides a systematic method for evaluating the processing of spatio - temporal information, providing a new way to study the response of models to different spatio - temporal combinations.
- Experimental results show that the TIME layer can improve the recognition accuracy of the model, especially when dealing with complex actions.

### Summary
The main goal of the paper is to solve the deficiencies of existing video action recognition methods in integrating spatial and temporal information by introducing the TIME layer, thereby improving the recognition accuracy of the model. The TIME layer generates a new frame structure by rearranging video frames, which retains both temporal dynamics and spatial details, providing an effective solution for video action recognition tasks.