Joint one-sided synthetic unpaired image translation and segmentation for colorectal cancer prevention

Enric Moreu,Eric Arazo,Kevin McGuinness,Noel E. O'Connor
DOI: https://doi.org/10.1111/exsy.13137
2023-07-21
Abstract:Deep learning has shown excellent performance in analysing medical images. However, datasets are difficult to obtain due privacy issues, standardization problems, and lack of annotations. We address these problems by producing realistic synthetic images using a combination of 3D technologies and generative adversarial networks. We propose CUT-seg, a joint training where a segmentation model and a generative model are jointly trained to produce realistic images while learning to segment polyps. We take advantage of recent one-sided translation models because they use significantly less memory, allowing us to add a segmentation model in the training loop. CUT-seg performs better, is computationally less expensive, and requires less real images than other memory-intensive image translation approaches that require two stage training. Promising results are achieved on five real polyp segmentation datasets using only one real image and zero real annotations. As a part of this study we release Synth-Colon, an entirely synthetic dataset that includes 20000 realistic colon images and additional details about depth and 3D geometry: <a class="link-external link-https" href="https://enric1994.github.io/synth-colon" rel="external noopener nofollow">this https URL</a>
Computer Vision and Pattern Recognition,Artificial Intelligence
What problem does this paper attempt to address?
The paper aims to address the issue of polyp segmentation in colorectal cancer prevention, specifically by utilizing synthetic data to train polyp segmentation models without the need for manual annotation. The main objectives of the study include: 1. **Overcoming Data Limitations**: Due to privacy concerns, standardization challenges, and the lack of expert annotators, real medical image datasets are difficult to obtain. The paper proposes a method that combines 3D rendering techniques and Generative Adversarial Networks (GAN) to generate realistic synthetic images and their automatic annotations. 2. **Joint Training Model**: The CUT-seg model is proposed, which is an end-to-end approach that can simultaneously perform image translation (from synthetic domain to real domain) and polyp segmentation tasks. Compared to traditional two-stage methods, this approach improves computational efficiency and performance. 3. **Reducing Dependence on Real Data**: Experiments show that even with only a small amount of real images (e.g., 1 image), CUT-seg can achieve good segmentation results, thereby reducing the need for a large amount of real image data. 4. **Releasing Synthetic Dataset**: The Synth-Colon dataset is released, which is the first large-scale dataset composed entirely of synthetic images, containing 20,000 realistic colon images along with corresponding depth information and 3D geometric structures to support future research work. In summary, this paper aims to solve the data acquisition challenges in medical image segmentation through synthetic data and technical means, and to improve the practicality and generalization ability of the models.