Abstract:The fusion of optical coherence tomography (OCT) and fundus modality information can provide a comprehensive diagnosis for retinal artery occlusion (RAO) disease, where OCT provides the cross-sectional examination of the fundus image. Given multi-modal retinal images, an anomaly diagnosis model can discriminate RAO without the need for real diseased samples. Despite this, previous studies have only focused on single-modal diagnosis, because of: 1) the lack of paired modality samples; and 2) the significant imaging differences, which make the fusion difficult with small-scale medical data. In this paper, we describe how we first built a multi-modal RAO dataset including both OCT and fundus modalities, which supports both the anomaly detection and localization tasks with pixel-level annotation. Motivated by the powerful generalization ability of the recent visual foundation model known as the Segment Anything Model (SAM), we adapted it for our task considering the small-scale property of retinal samples. Specifically, a modality-shared decoder with task-specific tokens is introduced to make SAM support the multi-modal image setting, which includes a mask token for the anomaly localization task at the pixel level and a fusion token for the anomaly detection task at the case level. Since SAM has little medical knowledge and lacks the learning of the “normal” concept, it is infeasible to localize RAO anomalies in the zero-shot manner. To integrate expert retinal knowledge while keeping the general segmentation knowledge, general anomaly simulation for both modalities and a low-level prompt-tuning strategy are introduced. The experiments conducted in this study show that the adapted model can surpass the state-of-the-art model by a large margin. This study sets the first benchmark for the multi-modal anomaly detection and localization tasks in the medical community. The code is available at https://github.com/Jingtao-Li-CVer/MMRAD.

A Collaborative Anomaly Localization Method Based on Multi-Modal Images

Adapting the Segment Anything Model for Multi-modal Retinal Anomaly Detection and Localization

Multimodal Industrial Anomaly Detection via Hybrid Fusion

Collaborative Discrepancy Optimization for Reliable Image Anomaly Localization

A Diffusion-Based Framework for Multi-Class Anomaly Detection

Complementary Pseudo Multimodal Feature for Point Cloud Anomaly Detection

Pull & Push: Leveraging Differential Knowledge Distillation for Efficient Unsupervised Anomaly Detection and Localization

ALMRR: Anomaly Localization Mamba on Industrial Textured Surface with Feature Reconstruction and Refinement

Multimodal Industrial Anomaly Detection by Crossmodal Feature Mapping

A Model-Agnostic Framework for Universal Anomaly Detection of Multi-organ and Multi-modal Images

DAS3D: Dual-modality Anomaly Synthesis for 3D Anomaly Detection

A Novel MAE-Based Self-Supervised Anomaly Detection and Localization Method

AnomalyNCD: Towards Novel Anomaly Class Discovery in Industrial Scenarios

Uni-3DAD: GAN-Inversion Aided Universal 3D Anomaly Detection on Model-free Products

Towards Zero-shot 3D Anomaly Localization

Focus Your Distribution: Coarse-to-Fine Non-Contrastive Learning for Anomaly Detection and Localization

Incomplete Multimodal Industrial Anomaly Detection via Cross-Modal Distillation

Enhanced multi-scale features mutual mapping fusion based on reverse knowledge distillation for industrial anomaly detection and localization

Anomaly Detection Based on Spatio-Temporal Sparse Representation and Visual Attention Analysis.

Memoryless Multimodal Anomaly Detection via Student–Teacher Network and Signed Distance Learning

Dual-Branch Reconstruction Network for Industrial Anomaly Detection with RGB-D Data