Abstract:Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing polyps (precursors of colon cancer). The non-invasive VC is normally used to inspect a 3D reconstructed colon (from CT scans) for polyps and if found, the OC procedure is performed to physically traverse the colon via endoscope and remove these polyps. In this paper, we present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between OC and VC to augment OC video sequences with scale-consistent depth information from VC, and augment VC with patient-specific textures, color and specular highlights from OC (e.g, for realistic polyp synthesis). Both OC and VC contain structural information, but it is obscured in OC by additional patient-specific texture and specular highlights, hence making the translation from OC to VC lossy. The existing CycleGAN approaches do not handle lossy transformations. To address this shortcoming, we introduce an extended cycle consistency loss, which compares the geometric structures from OC in the VC domain. This loss removes the need for the CycleGAN to embed OC information in the VC domain. To handle a stronger removal of the textures and lighting, a Directional Discriminator is introduced to differentiate the direction of translation (by creating paired information for the discriminator), as opposed to the standard CycleGAN which is direction-agnostic. Combining the extended cycle consistency loss and the Directional Discriminator, we show state-of-the-art results on scale-consistent depth inference for phantom, textured VC and for real polyp and normal colon video sequences. We also present results for realistic pendunculated and flat polyp synthesis from bumps introduced in 3D VC models. Code/models: <a class="link-external link-https" href="https://github.com/nadeemlab/CEP" rel="external noopener nofollow">this https URL</a>.

RT-GAN: Recurrent Temporal GAN for Adding Lightweight Temporal Consistency to Frame-Based Domain Translation Approaches

Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translation

CLTS-GAN: Color-Lighting-Texture-Specular Reflection Augmentation for Colonoscopy

Generative Adversarial Networks for Video-to-Video Domain Adaptation.

Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries

Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data

MedGAN: Medical Image Translation using GANs

LC-GAN: Image-to-image Translation Based on Generative Adversarial Network for Endoscopic Images

TR-TransGAN: Temporal Recurrent Transformer Generative Adversarial Network for Longitudinal MRI Dataset Expansion

BioGAN: An unpaired GAN-based image to image translation model for microbiological images

AV-GAN: Attention-Based Varifocal Generative Adversarial Network for Uneven Medical Image Translation

Using Conditional Generative Adversarial Networks to Reduce the Effects of Latency in Robotic Telesurgery

Memory-efficient GAN-based domain translation of high resolution 3D medical images

Enhancing GANs with Contrastive Learning-Based Multistage Progressive Finetuning SNN and RL-Based External Optimization

Biphasic Learning of GANs for High-Resolution Image-to-Image Translation

ARA-GAN: Adaptive Residual Attention Generative Adversarial Network for Retinal Vessel Segmentation

Recycle-GAN: Unsupervised Video Retargeting

SA-GAN: Structure-Aware GAN for Organ-Preserving Synthetic CT Generation

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its Effect on Image Correspondence

Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization