Abstract:The fusion of optical coherence tomography (OCT) and fundus modality information can provide a comprehensive diagnosis for retinal artery occlusion (RAO) disease, where OCT provides the cross-sectional examination of the fundus image. Given multi-modal retinal images, an anomaly diagnosis model can discriminate RAO without the need for real diseased samples. Despite this, previous studies have only focused on single-modal diagnosis, because of: 1) the lack of paired modality samples; and 2) the significant imaging differences, which make the fusion difficult with small-scale medical data. In this paper, we describe how we first built a multi-modal RAO dataset including both OCT and fundus modalities, which supports both the anomaly detection and localization tasks with pixel-level annotation. Motivated by the powerful generalization ability of the recent visual foundation model known as the Segment Anything Model (SAM), we adapted it for our task considering the small-scale property of retinal samples. Specifically, a modality-shared decoder with task-specific tokens is introduced to make SAM support the multi-modal image setting, which includes a mask token for the anomaly localization task at the pixel level and a fusion token for the anomaly detection task at the case level. Since SAM has little medical knowledge and lacks the learning of the “normal” concept, it is infeasible to localize RAO anomalies in the zero-shot manner. To integrate expert retinal knowledge while keeping the general segmentation knowledge, general anomaly simulation for both modalities and a low-level prompt-tuning strategy are introduced. The experiments conducted in this study show that the adapted model can surpass the state-of-the-art model by a large margin. This study sets the first benchmark for the multi-modal anomaly detection and localization tasks in the medical community. The code is available at https://github.com/Jingtao-Li-CVer/MMRAD.

SAM-OCTA: A Fine-Tuning Strategy for Applying Foundation Model to OCTA Image Segmentation Tasks

SAM-OCTA2: Layer Sequence OCTA Segmentation with Fine-tuned Segment Anything Model 2

SAM-OCTA: Prompting Segment-Anything for OCTA Image Segmentation

Integrating Spatial Prior Adapter for Enhancing SAM Performance in Medical Image Segmentation

Adapting the Segment Anything Model for Multi-modal Retinal Anomaly Detection and Localization

SAMedOCT: Adapting Segment Anything Model (SAM) for Retinal OCT

OCTAMamba: A State-Space Model Approach for Precision OCTA Vasculature Segmentation

High Performance OCTA Enabled by Combining Features of Shape, Intensity, and Complex Decorrelation

Ladder Fine-tuning approach for SAM integrating complementary network

Adapting SAM for Foveal Avascular Zone Segmentation in Retinal Optical Coherence Tomography Angiography Images

An Accurate and Efficient Neural Network for OCTA Vessel Segmentation and a New Dataset

Multi-Prompt Fine-Tuning of Foundation Models for Enhanced Medical Image Segmentation

OCTA-500: A Retinal Dataset for Optical Coherence Tomography Angiography Study

Segmentation of Low-Light Optical Coherence Tomography Angiography Images under the Constraints of Vascular Network Topology

OSAM-Fundus: A Training-Free, One-Shot Segmentation Framework for Optic Disc and Cup in Fundus Images

S-SAM: SVD-based Fine-Tuning of Segment Anything Model for Medical Image Segmentation

Novel OCT mosaicking pipeline with Feature- and Pixel-based registration

A Deep Learning-based Quality Assessment and Segmentation System with a Large-scale Benchmark Dataset for Optical Coherence Tomographic Angiography Image

OMSN and FAROS: OCTA Microstructure Segmentation Network and Fully Annotated Retinal OCTA Segmentation Dataset

How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model

Concatenate, Fine-tuning, Re-training: A SAM-enabled Framework for Semi-supervised 3D Medical Image Segmentation