OTRE: Where Optimal Transport Guided Unpaired Image-to-Image Translation Meets Regularization by Enhancing

Wenhui Zhu,Peijie Qiu,Oana M. Dumitrascu,Jacob M. Sobczak,Mohammad Farazi,Zhangsihao Yang,Keshav Nandakumar,Yalin Wang

DOI: https://doi.org/10.48550/arXiv.2302.03003

2023-04-09

Abstract:Non-mydriatic retinal color fundus photography (CFP) is widely available due to the advantage of not requiring pupillary dilation, however, is prone to poor quality due to operators, systemic imperfections, or patient-related causes. Optimal retinal image quality is mandated for accurate medical diagnoses and automated analyses. Herein, we leveraged the Optimal Transport (OT) theory to propose an unpaired image-to-image translation scheme for mapping low-quality retinal CFPs to high-quality counterparts. Furthermore, to improve the flexibility, robustness, and applicability of our image enhancement pipeline in the clinical practice, we generalized a state-of-the-art model-based image reconstruction method, regularization by denoising, by plugging in priors learned by our OT-guided image-to-image translation network. We named it as regularization by enhancing (RE). We validated the integrated framework, OTRE, on three publicly available retinal image datasets by assessing the quality after enhancement and their performance on various downstream tasks, including diabetic retinopathy grading, vessel segmentation, and diabetic lesion segmentation. The experimental results demonstrated the superiority of our proposed framework over some state-of-the-art unsupervised competitors and a state-of-the-art supervised method.

Image and Video Processing,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve the quality of non - mydriatic retinal color fundus photography (CFP). Specifically, due to operator, system defects or patient - related reasons, non - mydriatic retinal color fundus photography is prone to produce low - quality images, which will affect the accuracy of medical diagnosis and automatic analysis. For this reason, the author proposes an unpaired image - to - image conversion scheme based on the optimal transport theory (OT) to map low - quality retinal CFP to high - quality corresponding images. In addition, in order to improve the flexibility, robustness and applicability of the image enhancement pipeline in clinical practice, the author extends the model - based image reconstruction method - regularization by denoising (RED) by introducing prior knowledge learned from OT - guided image - to - image conversion network learning, and proposes a regularization by enhancing (RE) module. Finally, the author names this integrated framework OTRE and verifies its performance on three publicly available retinal image datasets, evaluating the enhanced image quality and performance on various downstream tasks (such as diabetic retinopathy grading, vessel segmentation and diabetic lesion segmentation).

The main contributions of the paper are as follows:
1. A new OT - based generative adversarial network (GAN) unsupervised end - to - end retinal image enhancement training scheme is proposed, and a maximum information - retaining consistency mechanism is adopted to prevent excessive tampering of lesions and structures.
2. An RE module is introduced to optimize the output of the OT module, improving the flexibility, robustness and practical application ability of the system.
3. Experimental results show that the proposed method outperforms unsupervised and state - of - the - art supervised methods in three large retinal imaging cohorts.