Abstract:ABSTRACT Histopathology, vital in diagnosing medical conditions, especially in cancer research, relies on analyzing histopathology images (HIs). Nuclei segmentation, a key task, involves precisely identifying cell nuclei boundaries. Manual segmentation by pathologists is time‐consuming, prompting the need for robust automated methods. Challenges in segmentation arise from HI complexities, necessitating advanced techniques. Recent advancements in deep learning, particularly Convolutional Neural Networks (CNNs), have transformed nuclei segmentation. This study emphasizes feature extraction, introducing the ConvNext Mixer‐based Encoder‐Decoder (CNM‐ED) model. Unlike traditional CNN based models, the proposed CNM‐ED model enables the extraction of spatial and long context features to address the inherent complexities of histopathology images. This method leverages a multi‐path strategy using a traditional CNN architecture as well as different paths focused on obtaining customized long context features using the ConvNext Mixer block structure that combines ConvMixer and ConvNext blocks. The fusion of these diverse features in the final segmentation output enables improved accuracy and performance, surpassing existing state‐of‐the‐art segmentation models. Moreover, our multi‐level feature extraction strategy is more effective than models using self‐attention mechanisms such as SwinUnet and TransUnet, which have been frequently used in recent years. Experimental studies were conducted using five different datasets (TNBC, MoNuSeg, CoNSeP, CPM17, and CryoNuSeg) to analyze the performance of the proposed CNM‐ED model. Comparisons were made with various CNN based models in the literature using evaluation metrics such as accuracy, AJI, macro F1 score, macro intersection over union, macro precision, and macro recall. It was observed that the proposed CNM‐ED model achieved highly successful results across all metrics. Through comparisons with state‐art‐of models from the literature, the proposed CNM‐ED model stands out as a promising advancement in nuclei segmentation, addressing the intricacies of histopathological images. The model demonstrates enhanced diagnostic capabilities and holds the potential for significant progress in medical research.

Cell Segmentation Using a Similarity Interface With a Multi-Task Convolutional Neural Network

Joint Feature Learning for Cell Segmentation Based on Multi-scale Convolutional U-Net.

Domain Adapted Multitask Learning for Segmenting Amoeboid Cells in Microscopy

Scellseg: a Style-Aware Cell Instance Segmentation Tool with Pre-Training and Contrastive Fine-Tuning

Learning to segment clustered amoeboid cells from brightfield microscopy via multi-task learning with adaptive weight selection

Advanced Multi-Microscopic Views Cell Semi-supervised Segmentation

DeepSplit: Segmentation of Microscopy Images Using Multi-task Convolutional Networks

Simultaneous Detection and Segmentation of Cell Nuclei Based on Convolutional Neural Network.

Learn to segment single cells with deep distance estimator and deep cell detector

CellSegNet: an Adaptive Multi-Resolution Hybrid Network for Cell Segmentation

Bend-Net: Bending Loss Regularized Multitask Learning Network for Nuclei Segmentation in Histopathology Images

Microscopy Cell Segmentation Via Convolutional LSTM Networks

A Reliable Distributed Convolutional Neural Network for Biology Image Segmentation.

Joint Cell Nuclei Detection and Segmentation in Microscopy Images Using 3D Convolutional Networks

A Multi-Task Cross-Task Learning Architecture for Ad-hoc Uncertainty Estimation in 3D Cardiac MRI Image Segmentation

The multimodality cell segmentation challenge: toward universal solutions

Celltrack R-CNN: A Novel End-To-End Deep Neural Network for Cell Segmentation and Tracking in Microscopy Images

ConvNext Mixer‐Based Encoder Decoder Method for Nuclei Segmentation in Histopathology Images

AL-Net: Attention Learning Network based on Multi-Task Learning for Cervical Nucleus Segmentation

Multi-stream Cell Segmentation with Low-level Cues for Multi-modality Images

Dual U-Net for the Segmentation of Overlapping Glioma Nuclei.