Abstract:The fusion of optical coherence tomography (OCT) and fundus modality information can provide a comprehensive diagnosis for retinal artery occlusion (RAO) disease, where OCT provides the cross-sectional examination of the fundus image. Given multi-modal retinal images, an anomaly diagnosis model can discriminate RAO without the need for real diseased samples. Despite this, previous studies have only focused on single-modal diagnosis, because of: 1) the lack of paired modality samples; and 2) the significant imaging differences, which make the fusion difficult with small-scale medical data. In this paper, we describe how we first built a multi-modal RAO dataset including both OCT and fundus modalities, which supports both the anomaly detection and localization tasks with pixel-level annotation. Motivated by the powerful generalization ability of the recent visual foundation model known as the Segment Anything Model (SAM), we adapted it for our task considering the small-scale property of retinal samples. Specifically, a modality-shared decoder with task-specific tokens is introduced to make SAM support the multi-modal image setting, which includes a mask token for the anomaly localization task at the pixel level and a fusion token for the anomaly detection task at the case level. Since SAM has little medical knowledge and lacks the learning of the “normal” concept, it is infeasible to localize RAO anomalies in the zero-shot manner. To integrate expert retinal knowledge while keeping the general segmentation knowledge, general anomaly simulation for both modalities and a low-level prompt-tuning strategy are introduced. The experiments conducted in this study show that the adapted model can surpass the state-of-the-art model by a large margin. This study sets the first benchmark for the multi-modal anomaly detection and localization tasks in the medical community. The code is available at https://github.com/Jingtao-Li-CVer/MMRAD.

A Model-Agnostic Framework for Universal Anomaly Detection of Multi-organ and Multi-modal Images

A Weakly-Supervised Anomaly Detection Method Via Adversarial Training for Medical Images

Adapting the Segment Anything Model for Multi-modal Retinal Anomaly Detection and Localization

Feasibility of Universal Anomaly Detection without Knowing the Abnormality in Medical Images

A Diffusion-Based Framework for Multi-Class Anomaly Detection

Anomaly Detection for Medical Images using Heterogeneous Auto-Encoder

Anomaly Detection in Chest X-rays Based on Dual-Attention Mechanism and Multi-Scale Feature Fusion

Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images

Anomaly Detection for Medical Images Using Self-Supervised and Translation-Consistent Features

Multi-modal vision-language model for generalizable annotation-free pathology localization and clinical diagnosis

DBTN: An adaptive neural network for multiple-disease detection via imbalanced medical images distribution

Hybrid Intelligent-Annotation Organ Segmentation on Medical Datasets

Multi-layer Aggregation as a key to feature-based OOD detection

A multi-scale anomaly detection framework for retinal OCT images based on the Bayesian neural network

Synomaly Noise and Multi-Stage Diffusion: A Novel Approach for Unsupervised Anomaly Detection in Ultrasound Imaging

Uncertainty-aware prototypical learning for anomaly detection in medical images

MedIAnomaly: A comparative study of anomaly detection in medical images

Modality-Agnostic Structural Image Representation Learning for Deformable Multi-Modality Medical Image Registration

A Unified Model for Multi-class Anomaly Detection

MSDAN: Multi-Scale Self-Attention Unsupervised Domain Adaptation Network for Thyroid Ultrasound Images

Individualized Statistical Modeling of Lesions in Fundus Images for Anomaly Detection