Abstract:Color fundus photography (CFP) and Optical coherence tomography (OCT) images are two of the most widely used modalities in the clinical diagnosis and management of retinal diseases. Despite the widespread use of multimodal imaging in clinical practice, few methods for automated diagnosis of eye diseases utilize correlated and complementary information from multiple modalities effectively. This paper explores how to leverage the information from CFP and OCT images to improve the automated diagnosis of retinal diseases. We propose a novel multimodal learning method, named geometric correspondence-based multimodal learning network (GeCoM-Net), to achieve the fusion of CFP and OCT images. Specifically, inspired by clinical observations, we consider the geometric correspondence between the OCT slice and the CFP region to learn the correlated features of the two modalities for robust fusion. Furthermore, we design a new feature selection strategy to extract discriminative OCT representations by automatically selecting the important feature maps from OCT slices. Unlike the existing multimodal learning methods, GeCoM-Net is the first method that formulates the geometric relationships between the OCT slice and the corresponding region of the CFP image explicitly for CFP and OCT fusion. Experiments have been conducted on a large-scale private dataset and a publicly available dataset to evaluate the effectiveness of GeCoM-Net for diagnosing diabetic macular edema (DME), impaired visual acuity (VA) and glaucoma. The empirical results show that our method outperforms the current state-of-the-art multimodal learning methods by improving the AUROC score 0.4%, 1.9% and 2.9% for DME, VA and glaucoma detection, respectively.

Multiple ocular diseases detection by graph regularized multi-label learning

Multiple Ocular Diseases Classification With Graph Regularized Probabilistic Multi-Label Learning

Ocular Disease Detection from Multiple Informatics Domains.

Multi-label classification of fundus images with graph convolutional network and LightGBM

Diagnosis of Multiple Fundus Disorders Amidst a Scarcity of Medical Experts Via Self-supervised Machine Learning

Federated Ophthalmic Disease Diagnosis Based on Weighted Averaging and Local Optimization

Parallel Multi-Path Network for Ocular Disease Detection Inspired by Visual Cognition Mechanism

Rotation-Oriented Collaborative Self-Supervised Learning for Retinal Disease Diagnosis

Label-Aware Dual Graph Neural Networks for Multi-Label Fundus Image Classification

Multi-Modal Graph Learning for Disease Prediction

Synergic Adversarial Label Learning for Grading Retinal Diseases via Knowledge Distillation and Multi-Task Learning

Glaucoma Detection by Learning from Multiple Informatics Domains

Multi-step framework for glaucoma diagnosis in retinal fundus images using deep learning

Ophthalmic Disease Detection via Deep Learning With a Novel Mixture Loss Function

Multi-label classification of retinal diseases based on fundus images using Resnet and Transformer

Geometric Correspondence-Based Multimodal Learning for Ophthalmic Image Analysis

Discriminative ensemble meta-learning with co-regularization for rare fundus diseases diagnosis

Prediction of Different Eye Diseases Based on Fundus Photography via Deep Transfer Learning

Comparative Study of Transfer Learning Models for Retinal Disease Diagnosis from Fundus Images

A deep-learning pipeline for the diagnosis and grading of common blinding ophthalmic diseases based on lesion-focused classification model

Adaptive Multiscale Retinal Diagnosis: A Hybrid Trio-Model Approach for Comprehensive Fundus Multi-Disease Detection Leveraging Transfer Learning and Siamese Networks