Abstract:ABSTRACT Histopathology, vital in diagnosing medical conditions, especially in cancer research, relies on analyzing histopathology images (HIs). Nuclei segmentation, a key task, involves precisely identifying cell nuclei boundaries. Manual segmentation by pathologists is time‐consuming, prompting the need for robust automated methods. Challenges in segmentation arise from HI complexities, necessitating advanced techniques. Recent advancements in deep learning, particularly Convolutional Neural Networks (CNNs), have transformed nuclei segmentation. This study emphasizes feature extraction, introducing the ConvNext Mixer‐based Encoder‐Decoder (CNM‐ED) model. Unlike traditional CNN based models, the proposed CNM‐ED model enables the extraction of spatial and long context features to address the inherent complexities of histopathology images. This method leverages a multi‐path strategy using a traditional CNN architecture as well as different paths focused on obtaining customized long context features using the ConvNext Mixer block structure that combines ConvMixer and ConvNext blocks. The fusion of these diverse features in the final segmentation output enables improved accuracy and performance, surpassing existing state‐of‐the‐art segmentation models. Moreover, our multi‐level feature extraction strategy is more effective than models using self‐attention mechanisms such as SwinUnet and TransUnet, which have been frequently used in recent years. Experimental studies were conducted using five different datasets (TNBC, MoNuSeg, CoNSeP, CPM17, and CryoNuSeg) to analyze the performance of the proposed CNM‐ED model. Comparisons were made with various CNN based models in the literature using evaluation metrics such as accuracy, AJI, macro F1 score, macro intersection over union, macro precision, and macro recall. It was observed that the proposed CNM‐ED model achieved highly successful results across all metrics. Through comparisons with state‐art‐of models from the literature, the proposed CNM‐ED model stands out as a promising advancement in nuclei segmentation, addressing the intricacies of histopathological images. The model demonstrates enhanced diagnostic capabilities and holds the potential for significant progress in medical research.

Real-time and accurate deep learning-based multi-organ nucleus segmentation in histology images

High-resolution deep transferred ASPPU-Net for nuclei segmentation of histopathology images

Nucleus segmentation from the histopathological images of liver cancer through an efficient deep learning framework

A deep learning algorithm for one-step contour aware nuclei segmentation of histopathology images

A dual decoder U-Net-based model for nuclei instance segmentation in hematoxylin and eosin-stained histological images

HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images

Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images

ConvNext Mixer‐Based Encoder Decoder Method for Nuclei Segmentation in Histopathology Images

A Dual-Path Instance Segmentation Network Based on Nuclei Contour in Histology Image

Multi-Object Localization and Irrelevant-Semantic Separation for Nuclei Segmentation in Histopathology Images

Efficient Multi-Organ Multi-Center Cell Nuclei Segmentation Method Based on Deep Learnable Aggregation Network

DAN-NucNet: A Dual Attention Based Framework for Nuclei Segmentation in Cancer Histology Images under Wild Clinical Conditions

Efficient Nucleus Detection in Digital Pathology Images Using Multi-task and Multi-scale Instance Segmentation Network

Accurate Segmentation of Nuclei in Pathological Images Via Sparse Reconstruction and Deep Convolutional Networks

Nuclei probability and centroid map network for nuclei instance segmentation in histology images

Simultaneous Detection and Segmentation of Cell Nuclei Based on Convolutional Neural Network.

MDC-net: A new convolutional neural network for nucleus segmentation in histopathology images with distance maps and contour information

An Automated Method with Anchor-Free Detection and U-shaped Segmentation for Nuclei Instance Segmentation.

Robust nuclei segmentation in histopathology using ASPPU-Net and boundary refinement

Attention augmented distance regression and classification network for nuclei instance segmentation and type classification in histology images

DAN-NucNet++: A Bottom-Up Framework Using Dual Attention Network for Simultaneous Nuclei Segmentation and Classification in Digital Histology Images