Improving Nuclei Segmentation in Pathological Image Via Reinforcement Learning

Mengjiao Yao,Xiang Gao,Jiaxin Wang,Min Wang
DOI: https://doi.org/10.1109/mlccim55934.2022.00057
2022-01-01
Abstract:Nuclei segmentation of pathological images is an important task in computational pathology. Accurate segmentation results can help pathologists diagnose cancer more accurately. However, pathological images often exist on issues of cell aggregation, stacking, and uneven staining. Generally, we use the deep learning method to get relatively accurate segmentation results and then use the hard threshold with a probability of 0.5 as a means of foreground and background separation. While the hard threshold often leads to cell boundary loss and cell misclassification. To overcome these issues, we present an adaptive threshold segmentation approach based on reinforcement learning (RL), which takes the model output as an environment variable, and adaptively gives a threshold of foreground and background segmentation. Our proposed method uses Q- learning and Deep Q Network (DQN) algorithm in RL and analyzes their effects. The experimental results show that Q- learning and DQN have obtained 3.6% and 3.8% average IoU improvement respectively on the Multi-organ Nuclei Segmentation (MoNuSeg) dataset, compared to the hard threshold segmentation results. Furthermore, compared with other representative methods, our DQN based method obtains the state-of-the-art nuclei segmentation performance.
What problem does this paper attempt to address?