Abstract:Recent advancements in State Space Models, notably Mamba, have demonstrated superior performance over the dominant Transformer models, particularly in reducing the computational complexity from quadratic to linear. Yet, difficulties in adapting Mamba from language to vision tasks arise due to the distinct characteristics of visual data, such as the spatial locality and adjacency within images and large variations in information granularity across visual tokens. Existing vision Mamba approaches either flatten tokens into sequences in a raster scan fashion, which breaks the local adjacency of images, or manually partition tokens into windows, which limits their long-range modeling and generalization capabilities. To address these limitations, we present a new vision Mamba model, coined QuadMamba, that effectively captures local dependencies of varying granularities via quadtree-based image partition and scan. Concretely, our lightweight quadtree-based scan module learns to preserve the 2D locality of spatial regions within learned window quadrants. The module estimates the locality score of each token from their features, before adaptively partitioning tokens into window quadrants. An omnidirectional window shifting scheme is also introduced to capture more intact and informative features across different local regions. To make the discretized quadtree partition end-to-end trainable, we further devise a sequence masking strategy based on Gumbel-Softmax and its straight-through gradient estimator. Extensive experiments demonstrate that QuadMamba achieves state-of-the-art performance in various vision tasks, including image classification, object detection, instance segmentation, and semantic segmentation. The code is in <a class="link-external link-https" href="https://github.com/VISION-SJTU/QuadMamba" rel="external noopener nofollow">this https URL</a>.

MemoryMamba: Memory-Augmented State Space Model for Defect Recognition

Learning Quality-aware Dynamic Memory for Video Object Segmentation

Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion

Vehicle Recognition Model for Complex Scenarios Based on Human Memory Mechanism.

MDD-Net: A novel defect detection model of material microscope image

Clear Memory-Augmented Auto-Encoder for Surface Defect Detection

MSIDetector: Detecting Multi-Scenario Industrial Defects Using an Adapted Visual Foundation Model and Dual Thresholding Discriminator

Multi-modal background-aware for defect semantic segmentation with limited data

A Feature Memory Rearrangement Network for Visual Inspection of Textured Surface Defects Toward Edge Intelligent Manufacturing

QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space Model

Defect intelligent recognition of membrane product based on deep learning

Defect Detection of MEMS Based on Data Augmentation, WGAN-DIV-DC, and a YOLOv5 Model

VSSD: Vision Mamba with Non-Causal State Space Duality

Memory-Enhanced Spatial-Temporal Encoding Framework for Industrial Anomaly Detection System

A dynamic memory model for mechanical fault diagnosis using one-class support vector machine

Fab-ME: A Vision State-Space and Attention-Enhanced Framework for Fabric Defect Detection

Revealing and Mitigating the Local Pattern Shortcuts of Mamba

Latent Space Segmentation Model for Visual Surface Defect Inspection

PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition

RSMamba: Remote Sensing Image Classification with State Space Model