Abstract:In clinical research, the segmentation of irregularly shaped nuclei, particularly in mesenchymal areas like fibroblasts, is crucial yet often neglected. These irregular nuclei are significant for assessing tissue repair in immunotherapy, a process involving neovascularization and fibroblast proliferation. Proper segmentation of these nuclei is vital for evaluating immunotherapy’s efficacy, as it provides insights into pathological features. However, the challenge lies in the pronounced curvature variations of these non-convex nuclei, making their segmentation more difficult than that of regular nuclei. In this work, we introduce an undefined task to segment nuclei with both regular and irregular morphology, namely multi-shape nuclei segmentation. We propose a proposal-based method to perform multi-shape nuclei segmentation. By leveraging the two-stage structure of the proposal-based method, a powerful refinement module with high computational costs can be selectively deployed only in local regions, improving segmentation accuracy without compromising computational efficiency. We introduce a novel self-attention module to refine features in proposals for the sake of effectiveness and efficiency in the second stage. The self-attention module improves segmentation performance by capturing long-range dependencies to assist in distinguishing the foreground from the background. In this process, similar features get high attention weights while dissimilar ones get low attention weights. In the first stage, we introduce a residual attention module and a semantic-aware module to accurately predict candidate proposals. The two modules capture more interpretable features and introduce additional supervision through semantic-aware loss. In addition, we construct a dataset with a proportion of non-convex nuclei compared with existing nuclei datasets, namely the multi-shape nuclei (MsN) dataset. Our MSNSegNet method demonstrates notable improvements across various metrics compared to the second-highest-scoring methods. For all nuclei, the Dice$$\mathrm Dice$$ score improved by approximately 1.66%$$\%$$, AJI$$\mathrm AJI$$ by about 2.15%$$\%$$, and Diceobj$$\mathrm Dice_{obj}$$ by roughly 0.65%$$\%$$. For non-convex nuclei, which are crucial in clinical applications, our method’s AJI$$\mathrm AJI$$ improved significantly by approximately 3.86%$$\%$$ and Diceobj$$\mathrm Dice_{obj}$$ by around 2.54%$$\%$$. These enhancements underscore the effectiveness of our approach on multi-shape nuclei segmentation, particularly in challenging scenarios involving irregularly shaped nuclei.Graphical abstractThe framework of the MSNSegNet

Automated Segmentation Of Cervical Nuclei In Pap Smear Images Using Deformable Multi-Path Ensemble Model

Mutual-Complementing Framework for Nuclei Detection and Segmentation in Pathology Image

Multi-layer Segmentation Framework for Cell Nuclei Using Improved GVF Snake Model, Watershed, and Ellipse Fitting.

Segmenting Overlapping Cervical Cell In Pap Smear Images

Interacting Convolution with Pyramid Structure Network for Automated Segmentation of Cervical Nuclei in Pap Smear Images.

Cervical Nuclei Segmentation in Whole Slide Histopathology Images Using Convolution Neural Network

Accurate Segmentation of Nuclei in Pathological Images Via Sparse Reconstruction and Deep Convolutional Networks

Automatic cell nuclei segmentation and classification of cervical Pap smear images

SEENS: Nuclei Segmentation in Pap Smear Images with Selective Edge Enhancement

Accurate Segmentation of Cervical Cytoplasm and Nuclei Based on Multi-scale Convolutional Network and Graph Partitioning.

A Practical Segmentation Method for Automated Screening of Cervical Cytology

Deformable multi-level feature network applied to nucleus segmentation

Accurate segmentation of overlapping cells in cervical cytology with deep convolutional neural networks

Automatic Cytoplasm and Nuclei Segmentation for Color Cervical Smear Image Using an Efficient Gap-Search MRF

Selective Detection and Segmentation of Cervical Cells

Nuclei Segmentation in Histopathology Images Using Rotation Equivariant and Multi-level Feature Aggregation Neural Network.

An interactive nuclei segmentation framework with Voronoi diagrams and weighted convex difference for cervical cancer pathology images

Semi-supervised nuclei segmentation based on multi-edge features fusion attention network

Segmentation of Overlapped Cervical Cells Using Asymmetric Mixture Model and Shape Constraint Level Set Method

MSNSegNet: attention-based multi-shape nuclei instance segmentation in histopathology images

An automatic segmentation method for multispectral microscopic cervical cell images