Abstract:Objective: As an extension of optical coherence tomography (OCT), optical coherence tomographic angiography (OCTA) provides information on the blood flow status at the microlevel and is sensitive to changes in the fundus vessels. However, due to the distinct imaging mechanism of OCTA, existing models, which are primarily used for analyzing fundus images, do not work well on OCTA images. Effectively extracting and analyzing the information in OCTA images remains challenging. To this end, a deep learning framework that fuses multilevel information in OCTA images is proposed in this study. The effectiveness of the proposed model was demonstrated in the task of diabetic retinopathy (DR) classification. Method: First, a U-Net-based segmentation model was proposed to label the boundaries of large retinal vessels and the foveal avascular zone (FAZ) in OCTA images. Then, we designed an isolated concatenated block (ICB) structure to extract and fuse information from the original OCTA images and segmentation results at different fusion levels. Results: The experiments were conducted on 301 OCTA images. Of these images, 244 were labeled by ophthalmologists as normal images, and 57 were labeled as DR images. An accuracy of 93.1% and a mean intersection over union (mIOU) of 77.1% were achieved using the proposed large vessel and FAZ segmentation model. In the ablation experiment with 6-fold validation, the proposed deep learning framework that combines the proposed isolated and concatenated convolution process significantly improved the DR diagnosis accuracy. Moreover, inputting the merged images of the original OCTA images and segmentation results further improved the model performance. Finally, a DR diagnosis accuracy of 88.1% (95%CI ± 3.6%) and an area under the curve (AUC) of 0.92 were achieved using our proposed classification model, which significantly outperforms the state-of-the-art classification models. As a comparison, an accuracy of 83.7 (95%CI ± 1.5%) and AUC of 0.76 were obtained using EfficientNet. Significance. The visualization results show that the FAZ and the vascular region close to the FAZ provide more information for the model than the farther surrounding area. Furthermore, this study demonstrates that a clinically sophisticated designed deep learning model is not only able to effectively assist in the diagnosis but also help to locate new indicators for certain illnesses.

Leveraging Multimodal Fusion for Enhanced Diagnosis of Multiple Retinal Diseases in Ultra-wide OCTA

Adapting the Segment Anything Model for Multi-modal Retinal Anomaly Detection and Localization

MultiEYE: Dataset and Benchmark for OCT-Enhanced Retinal Disease Recognition from Fundus Images

Multi-Modal Multi-Instance Learning for Retinal Disease Recognition

Multimodal Information Fusion for Glaucoma and DR Classification

Geometric Correspondence-Based Multimodal Learning for Ophthalmic Image Analysis

OCTCube-M: A 3D multimodal optical coherence tomography foundation model for retinal and systemic diseases with cross-cohort and cross-device validation

Improved Automatic Diabetic Retinopathy Severity Classification Using Deep Multimodal Fusion of UWF-CFP and OCTA Images

Mstnet: method for glaucoma grading based on multimodal feature fusion of spatial relations

Multi-OCT-SelfNet: Integrating Self-Supervised Learning with Multi-Source Data Fusion for Enhanced Multi-Class Retinal Disease Classification

Towards multi-center glaucoma OCT image screening with semi-supervised joint structure and function multi-task learning

Parallel Multi-Path Network for Ocular Disease Detection Inspired by Visual Cognition Mechanism

Fusion of Multi-Source Retinal Fundus Images Via Automatic Registration for Clinical Diagnosis

Fundus-Enhanced Disease-Aware Distillation Model for Retinal Disease Classification from OCT Images

Unveiling the Power of High-Quality OCT: an Effective Fundus-based Modality Fusion Network for Fovea Localization

Diagnosing Diabetic Retinopathy in OCTA Images Based on Multilevel Information Fusion Using a Deep Learning Framework

Retinal Structure Detection in OCTA Image via Voting-Based Multitask Learning

Multi-View-Based Automatic Aided Diagnosis Method for Screening Multiple Diseases in Retinal OCT Images

Representation, Alignment, Fusion: A Generic Transformer-Based Framework for Multi-modal Glaucoma Recognition

Automated diagnosis of age‐related macular degeneration using multi‐modal vertical plane feature fusion via deep learning

A Deep Learning Analysis Framework for Ophthalmic Diseases and Physical Health from Binocular Fundus Image Pairs