Abstract:In clinical research, the segmentation of irregularly shaped nuclei, particularly in mesenchymal areas like fibroblasts, is crucial yet often neglected. These irregular nuclei are significant for assessing tissue repair in immunotherapy, a process involving neovascularization and fibroblast proliferation. Proper segmentation of these nuclei is vital for evaluating immunotherapy’s efficacy, as it provides insights into pathological features. However, the challenge lies in the pronounced curvature variations of these non-convex nuclei, making their segmentation more difficult than that of regular nuclei. In this work, we introduce an undefined task to segment nuclei with both regular and irregular morphology, namely multi-shape nuclei segmentation. We propose a proposal-based method to perform multi-shape nuclei segmentation. By leveraging the two-stage structure of the proposal-based method, a powerful refinement module with high computational costs can be selectively deployed only in local regions, improving segmentation accuracy without compromising computational efficiency. We introduce a novel self-attention module to refine features in proposals for the sake of effectiveness and efficiency in the second stage. The self-attention module improves segmentation performance by capturing long-range dependencies to assist in distinguishing the foreground from the background. In this process, similar features get high attention weights while dissimilar ones get low attention weights. In the first stage, we introduce a residual attention module and a semantic-aware module to accurately predict candidate proposals. The two modules capture more interpretable features and introduce additional supervision through semantic-aware loss. In addition, we construct a dataset with a proportion of non-convex nuclei compared with existing nuclei datasets, namely the multi-shape nuclei (MsN) dataset. Our MSNSegNet method demonstrates notable improvements across various metrics compared to the second-highest-scoring methods. For all nuclei, the Dice$$\mathrm Dice$$ score improved by approximately 1.66%$$\%$$, AJI$$\mathrm AJI$$ by about 2.15%$$\%$$, and Diceobj$$\mathrm Dice_{obj}$$ by roughly 0.65%$$\%$$. For non-convex nuclei, which are crucial in clinical applications, our method’s AJI$$\mathrm AJI$$ improved significantly by approximately 3.86%$$\%$$ and Diceobj$$\mathrm Dice_{obj}$$ by around 2.54%$$\%$$. These enhancements underscore the effectiveness of our approach on multi-shape nuclei segmentation, particularly in challenging scenarios involving irregularly shaped nuclei.Graphical abstractThe framework of the MSNSegNet

AL-Net: Attention Learning Network based on Multi-Task Learning for Cervical Nucleus Segmentation

Nuclei Segmentation Using Cascaded Bilateral Attention U-Net.

Enhancing Nucleus Segmentation with HARU-Net: A Hybrid Attention Based Residual U-Blocks Network

Multifunctional aggregation network of cell nuclei segmentation aiming histopathological diagnosis assistance: A new MA-Net construction

GCP-Net: A Gating Context-Aware Pooling Network for Cervical Cell Nuclei Segmentation

DSCA-Net: Double-stage Codec Attention Network for automatic nuclear segmentation

Dual U-Net for the Segmentation of Overlapping Glioma Nuclei.

W-Net: A Two-Stage Convolutional Network for Nucleus Detection in Histopathology Image

Convolutional Blur Attention Network for Cell Nuclei Segmentation

FRE-Net: Full-region enhanced network for nuclei segmentation in histopathology images

2MSPK-Net: A nuclei segmentation network based on multi-scale, multi-dimensional attention, and SAM prior knowledge

Real-time and accurate deep learning-based multi-organ nucleus segmentation in histology images

MSNSegNet: attention-based multi-shape nuclei instance segmentation in histopathology images

Efficient Multi-Organ Multi-Center Cell Nuclei Segmentation Method Based on Deep Learnable Aggregation Network

Multi-Scale TransUnet Combined with CBAM for Nuclear Image Segmentation.

LFANet: Lightweight feature attention network for abnormal cell segmentation in cervical cytology images

Nucleus segmentation of cervical cytology images based on multi-scale fuzzy clustering algorithm

AugMS-Net:Augmented Multiscale Network for Small Cervical Tumor Segmentation from MRI Volumes

A dual decoder U-Net-based model for nuclei instance segmentation in hematoxylin and eosin-stained histological images

Bend-Net: Bending Loss Regularized Multitask Learning Network for Nuclei Segmentation in Histopathology Images

MIU-Net: MIX-Attention and Inception U-Net for Histopathology Image Nuclei Segmentation