Abstract:Objective:The objective of this study is to develop an efficient multimodal learning framework for the classification of glaucoma. Glaucoma is a group of eye diseases that can result in vision loss and blindness, often due to delayed detection and treatment. Fundus images and optical coherence tomography (OCT) images have proven valuable for the diagnosis and management of glaucoma. However, current models that combine features from both modalities often lack efficient spatial relationship modeling. Approach:In this study, we propose an innovative approach to address the classification of glaucoma. We focus on leveraging the features of OCT volumes and harness the capabilities of transformer models to capture long-range spatial relationships. To achieve this, we introduce a 3D transformer model to extract features from OCT volumes, enhancing the model's effectiveness. Additionally, we employ downsampling techniques to enhance model efficiency. We then utilize the spatial feature relationships between OCT volumes and fundus images to fuse the features extracted from both sources. Main Results:Our proposed framework has yielded remarkable results, particularly in terms of glaucoma grading performance. We conducted our experiments using the GAMMA dataset, and our approach outperformed traditional feature fusion methods. By effectively modeling spatial relationships and combining OCT volume and fundus map features, our framework achieved outstanding classification results.Significance:This research is of significant importance in the field of glaucoma diagnosis and management. Efficient and accurate glaucoma classification is essential for timely intervention and prevention of vision loss. Our proposed approach, which integrates 3D transformer models, offers a novel way to extract and fuse features from OCT volumes and fundus images, ultimately enhancing the effectiveness of glaucoma classification. This work has the potential to contribute to improved patient care, particularly in the early detection and treatment of glaucoma, thereby reducing the risk of vision impairment and blindness.

Transformer-based Cross-Modal Multi-Contrast Network for Ophthalmic Diseases Diagnosis

Geometric Correspondence-Based Multimodal Learning for Ophthalmic Image Analysis

Parallel Multi-Path Network for Ocular Disease Detection Inspired by Visual Cognition Mechanism

Cross-Fundus Transformer for Multi-modal Diabetic Retinopathy Grading with Cataract

Mstnet: method for glaucoma grading based on multimodal feature fusion of spatial relations

Multi-label classification of retinal diseases based on fundus images using Resnet and Transformer

HTC-retina: A hybrid retinal diseases classification model using transformer-Convolutional Neural Network from optical coherence tomography images

Model long-range dependencies for multi-modality and multi-view retinopathy diagnosis through transformers

MultiEYE: Dataset and Benchmark for OCT-Enhanced Retinal Disease Recognition from Fundus Images

Ophthalmic Biomarker Detection with Parallel Prediction of Transformer and Convolutional Architecture

Multi-label ocular disease classification with a dense correlation deep neural network

Multi-resolution visual Mamba with multi-directional selective mechanism for retinal disease detection

Towards more efficient ophthalmic disease classification and lesion location via convolution transformer

Confidence-aware multi-modality learning for eye disease screening

A Health Diagnosis System based on Transfer Learning and Multi-scale Dilated Convolution from Binocular Fundus Image Pairs

TL-CCL: Two-level causal contrastive learning for multi-label ocular disease diagnosis with fundus images

Multi-OCT-SelfNet: Integrating Self-Supervised Learning with Multi-Source Data Fusion for Enhanced Multi-Class Retinal Disease Classification

CTT-Net: A Multi-view Cross-token Transformer for Cataract Postoperative Visual Acuity Prediction

HCTNet: A Hybrid ConvNet-Transformer Network for Retinal Optical Coherence Tomography Image Classification

A Classification Network for Ocular Diseases Based on Structure Feature and Visual Attention

Comparative Study of Transfer Learning Models for Retinal Disease Diagnosis from Fundus Images