Abstract:In clinical research, the segmentation of irregularly shaped nuclei, particularly in mesenchymal areas like fibroblasts, is crucial yet often neglected. These irregular nuclei are significant for assessing tissue repair in immunotherapy, a process involving neovascularization and fibroblast proliferation. Proper segmentation of these nuclei is vital for evaluating immunotherapy’s efficacy, as it provides insights into pathological features. However, the challenge lies in the pronounced curvature variations of these non-convex nuclei, making their segmentation more difficult than that of regular nuclei. In this work, we introduce an undefined task to segment nuclei with both regular and irregular morphology, namely multi-shape nuclei segmentation. We propose a proposal-based method to perform multi-shape nuclei segmentation. By leveraging the two-stage structure of the proposal-based method, a powerful refinement module with high computational costs can be selectively deployed only in local regions, improving segmentation accuracy without compromising computational efficiency. We introduce a novel self-attention module to refine features in proposals for the sake of effectiveness and efficiency in the second stage. The self-attention module improves segmentation performance by capturing long-range dependencies to assist in distinguishing the foreground from the background. In this process, similar features get high attention weights while dissimilar ones get low attention weights. In the first stage, we introduce a residual attention module and a semantic-aware module to accurately predict candidate proposals. The two modules capture more interpretable features and introduce additional supervision through semantic-aware loss. In addition, we construct a dataset with a proportion of non-convex nuclei compared with existing nuclei datasets, namely the multi-shape nuclei (MsN) dataset. Our MSNSegNet method demonstrates notable improvements across various metrics compared to the second-highest-scoring methods. For all nuclei, the Dice$$\mathrm Dice$$ score improved by approximately 1.66%$$\%$$, AJI$$\mathrm AJI$$ by about 2.15%$$\%$$, and Diceobj$$\mathrm Dice_{obj}$$ by roughly 0.65%$$\%$$. For non-convex nuclei, which are crucial in clinical applications, our method’s AJI$$\mathrm AJI$$ improved significantly by approximately 3.86%$$\%$$ and Diceobj$$\mathrm Dice_{obj}$$ by around 2.54%$$\%$$. These enhancements underscore the effectiveness of our approach on multi-shape nuclei segmentation, particularly in challenging scenarios involving irregularly shaped nuclei.Graphical abstractThe framework of the MSNSegNet

An Image Processing Pipeline to Detect and Segment Nuclei in Muscle Fiber Microscopic Images.

A Computational Approach to Detect and Segment Cytoplasm in Muscle Fiber Images

Mutual-Complementing Framework for Nuclei Detection and Segmentation in Pathology Image

Automatic and unbiased segmentation and quantification of myofibers in skeletal muscle

An Automatic Method for Robust and Fast Cell Detection in Bright Field Images from High-Throughput Microscopy.

MyoV: a deep learning-based tool for the automated quantification of muscle fibers

Automatic segmentation and supervised learning-based selection of nuclei in cancer tissue images

Segmentation of Microscopy Data for finding Nuclei in Divergent Images

FiNuTyper: design and validation of an automated deep learning‐based platform for simultaneous fiber and nucleus type analysis in human skeletal muscle

A deep learning-based approach for fully automated segmentation and quantitative analysis of muscle fibers in pig skeletal muscle

MF-Net: Automated Muscle Fiber Segmentation From Immunofluorescence Images Using a Local-Global Feature Fusion Network

Clumped Nuclei Segmentation with Adjacent Point Match and Local Shape based Intensity Analysis for Overlapped Nuclei in Fluorescence In-Situ Hybridization Images

A Deep Learning Approach for Histology-Based Nucleus Segmentation and Tumor Microenvironment Characterization

Stain guided mean-shift filtering in automatic detection of human tissue nuclei

Deep Learning Model for Cell Nuclei Segmentation and Lymphocyte Identification in Whole Slide Histology Images

Automatic extraction and measurement of ultrasonic muscle morphological parameters based on multi-stage fusion and segmentation

Nuclei segmentation of HE stained histopathological images based on feature global delivery connection network

Automated Nuclear Lamina Network Recognition and Quantitative Analysis in Structured Illumination Super-Resolution Microscope Images Using a Gaussian Mixture Model and Morphological Processing

MSNSegNet: attention-based multi-shape nuclei instance segmentation in histopathology images

Introducing NCL-SM: A Fully Annotated Dataset of Images from Human Skeletal Muscle Biopsies

A Dataset and a Technique for Generalized Nuclear Segmentation for Computational Pathology