Abstract:Instance segmentation is a challenging task in computer vision, as it requires distinguishing objects and predicting dense areas. Currently, segmentation models based on complex designs and large parameters have achieved remarkable accuracy. However, from a practical standpoint, achieving a balance between accuracy and speed is even more desirable. To address this need, this paper presents ESAMask, a real-time segmentation model fused with efficient sparse attention, which adheres to the principles of lightweight design and efficiency. In this work, we propose several key contributions. Firstly, we introduce a dynamic and sparse Related Semantic Perceived Attention mechanism (RSPA) for adaptive perception of different semantic information of various targets during feature extraction. RSPA uses the adjacency matrix to search for regions with high semantic correlation of the same target, which reduces computational cost. Additionally, we design the GSInvSAM structure to reduce redundant calculations of spliced features while enhancing interaction between channels when merging feature layers of different scales. Lastly, we introduce the Mixed Receptive Field Context Perception Module (MRFCPM) in the prototype branch to enable targets of different scales to capture the feature representation of the corresponding area during mask generation. MRFCPM fuses information from three branches of global content awareness, large kernel region awareness, and convolutional channel attention to explicitly model features at different scales. Through extensive experimental evaluation, ESAMask achieves a mask AP of 45.4 at a frame rate of 45.2 FPS on the COCO dataset, surpassing current instance segmentation methods in terms of the accuracy–speed trade-off, as demonstrated by our comprehensive experimental results. In addition, the high-quality segmentation results of our proposed method for objects of various classes and scales can be intuitively observed from the visualized segmentation outputs.

ISDA: Position-Aware Instance Segmentation with Deformable Attention

Weakly Supervised Instance Segmentation Using Multi-Prior Fusion.

Dpnet: End-to-end Aerial Image Segmentation Via Deformable Point Network.

Dynamic Focus-aware Positional Queries for Semantic Segmentation

Supervised Edge Attention Network for Accurate Image Instance Segmentation

DB-BlendMask: Decomposed Attention and Balanced BlendMask for Instance Segmentation of High-Resolution Remote Sensing Images

BAISeg: Boundary Assisted Weakly Supervised Instance Segmentation

BoxInst: High-Performance Instance Segmentation with Box Annotations.

DAS: A Deformable Attention to Capture Salient Information in CNNs

HAISTA-NET: Human Assisted Instance Segmentation Through Attention

Position Attention Optimized Deep Semantic Segmentation

Boundary-Aware Instance Segmentation

Synthetic Instance Segmentation from Semantic Image Segmentation Masks

BorderPointsMask: One-stage Instance Segmentation with Boundary Points Representation.

CANDY: Category-Kernelized Dynamic Convolution for Instance Segmentation

ESAMask: Real-Time Instance Segmentation Fused with Efficient Sparse Attention

Regularized Densely-Connected Pyramid Network for Salient Instance Segmentation

DAABNet: depth-wise asymmetric attention bottleneck for real-time semantic segmentation

Fully Convolutional Instance-Aware Semantic Segmentation

NDAM-YOLOseg: a real-time instance segmentation model based on multi-head attention mechanism