Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

Lianghui Zhu,Bencheng Liao,Qian Zhang,Xinlong Wang,Wenyu Liu,Xinggang Wang

DOI: https://doi.org/10.48550/arXiv.2401.09417

2024-01-17

Computer Vision and Pattern Recognition

Abstract:Recently the state space models (SSMs) with efficient hardware-aware designs, i.e., the Mamba deep learning model, have shown great potential for long sequence modeling. Meanwhile building efficient and generic vision backbones purely upon SSMs is an appealing direction. However, representing visual data is challenging for SSMs due to the position-sensitivity of visual data and the requirement of global context for visual understanding. In this paper, we show that the reliance on self-attention for visual representation learning is not necessary and propose a new generic vision backbone with bidirectional Mamba blocks (Vim), which marks the image sequences with position embeddings and compresses the visual representation with bidirectional state space models. On ImageNet classification, COCO object detection, and ADE20k semantic segmentation tasks, Vim achieves higher performance compared to well-established vision transformers like DeiT, while also demonstrating significantly improved computation & memory efficiency. For example, Vim is 2.8$\times$ faster than DeiT and saves 86.8% GPU memory when performing batch inference to extract features on images with a resolution of 1248$\times$1248. The results demonstrate that Vim is capable of overcoming the computation & memory constraints on performing Transformer-style understanding for high-resolution images and it has great potential to be the next-generation backbone for vision foundation models. Code is available at https://github.com/hustvl/Vim.

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the following issues:

1. **Computational and Memory Efficiency in Visual Representation Learning**:
- Current visual transformers (e.g., ViT) face challenges in speed and memory usage due to the self-attention mechanism when processing high-resolution images.

2. **Capability of Pure State Space Models (SSM) in Visual Tasks**:
- Existing SSM methods perform well in handling long sequence data but lack positional awareness in visual tasks. Therefore, a new model is needed to overcome this limitation.

3. **Design of Efficient Visual Backbone Networks**:
- A new architecture based on bidirectional state space models, Vision Mamba (Vim), is proposed to achieve efficient and general visual representation learning. Vim not only outperforms existing visual transformers (e.g., DeiT) in performance but also has significant computational and memory efficiency advantages when processing high-resolution images.

Through these improvements, Vim demonstrates its potential in large-scale pre-training and downstream dense prediction tasks, making it a strong candidate for the next generation of foundational visual models.