VideoMamba: State Space Model for Efficient Video Understanding

Kunchang Li,Xinhao Li,Yi Wang,Yinan He,Yali Wang,Limin Wang,Yu Qiao

DOI: https://doi.org/10.48550/arXiv.2403.06977

2024-03-11

Computer Vision and Pattern Recognition

Abstract:Addressing the dual challenges of local redundancy and global dependencies in video understanding, this work innovatively adapts the Mamba to the video domain. The proposed VideoMamba overcomes the limitations of existing 3D convolution neural networks and video transformers. Its linear-complexity operator enables efficient long-term modeling, which is crucial for high-resolution long video understanding. Extensive evaluations reveal VideoMamba's four core abilities: (1) Scalability in the visual domain without extensive dataset pretraining, thanks to a novel self-distillation technique; (2) Sensitivity for recognizing short-term actions even with fine-grained motion differences; (3) Superiority in long-term video understanding, showcasing significant advancements over traditional feature-based models; and (4) Compatibility with other modalities, demonstrating robustness in multi-modal contexts. Through these distinct advantages, VideoMamba sets a new benchmark for video understanding, offering a scalable and efficient solution for comprehensive video understanding. All the code and models are available at https://github.com/OpenGVLab/VideoMamba.

What problem does this paper attempt to address?

This paper attempts to solve two main problems in video understanding: local redundancy and global dependencies. Specifically:

1. **Local Redundancy**: In short - time video clips, the information between adjacent frames is highly similar, resulting in a large amount of spatio - temporal redundancy.
2. **Global Dependencies**: There are complex spatio - temporal dependency relationships in long - time videos, which pose challenges to the long - term modeling ability of the model.

Although the existing 3D Convolutional Neural Networks (CNNs) and video Transformers perform well in dealing with local or global dependencies, it is difficult to deal with these two problems simultaneously. Especially for the understanding of long videos, the existing methods are usually computationally costly and inefficient.

To solve these problems, the paper introduces **VideoMamba**, a new video - understanding model based on the State - Space Model (SSM). The main contributions of VideoMamba include:

- **Linear - Complexity Operations**: Efficient long - term modeling is achieved through linear - complexity operations, which is crucial for the understanding of high - resolution long videos.
- **Self - Distillation Technique**: A simple self - distillation strategy is introduced, so that the model will not over - fit when expanding, and does not require pre - training on large - scale datasets.
- **Sensitivity to Short - Term Action Recognition**: It can accurately recognize short - term actions with subtle motion differences.
- **Multi - Modal Compatibility**: It shows strong performance in multi - modal tasks such as video - text retrieval.

Through these innovations, VideoMamba sets a new benchmark in the field of video understanding and provides a scalable and efficient solution.

### Formula Representation

The basic formulas of the State - Space Model (SSM) involved in the paper are as follows:

\[
h'(t)=A h(t)+B x(t)
\]
\[
y(t)=C h(t)
\]

where:
- \(h(t)\in\mathbb{R}^N\) is the hidden state,
- \(x(t)\in\mathbb{R}^L\) is the input sequence,
- \(y(t)\in\mathbb{R}^L\) is the output sequence,
- \(A\in\mathbb{R}^{N\times N}\) is the evolution matrix of the system,
- \(B\in\mathbb{R}^{N\times1}\) and \(C\in\mathbb{R}^{N\times1}\) are projection matrices.

The discretized formulas are:

\[
A_d = \exp(\Delta A)
\]
\[
B_d=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot\Delta B
\]
\[
h_t = A_d h_{t - 1}+B_d x_t
\]
\[
y_t = C h_t
\]

These formulas ensure the efficiency and accuracy of the model when processing long videos.