Abstract:The fusion of optical coherence tomography (OCT) and fundus modality information can provide a comprehensive diagnosis for retinal artery occlusion (RAO) disease, where OCT provides the cross-sectional examination of the fundus image. Given multi-modal retinal images, an anomaly diagnosis model can discriminate RAO without the need for real diseased samples. Despite this, previous studies have only focused on single-modal diagnosis, because of: 1) the lack of paired modality samples; and 2) the significant imaging differences, which make the fusion difficult with small-scale medical data. In this paper, we describe how we first built a multi-modal RAO dataset including both OCT and fundus modalities, which supports both the anomaly detection and localization tasks with pixel-level annotation. Motivated by the powerful generalization ability of the recent visual foundation model known as the Segment Anything Model (SAM), we adapted it for our task considering the small-scale property of retinal samples. Specifically, a modality-shared decoder with task-specific tokens is introduced to make SAM support the multi-modal image setting, which includes a mask token for the anomaly localization task at the pixel level and a fusion token for the anomaly detection task at the case level. Since SAM has little medical knowledge and lacks the learning of the “normal” concept, it is infeasible to localize RAO anomalies in the zero-shot manner. To integrate expert retinal knowledge while keeping the general segmentation knowledge, general anomaly simulation for both modalities and a low-level prompt-tuning strategy are introduced. The experiments conducted in this study show that the adapted model can surpass the state-of-the-art model by a large margin. This study sets the first benchmark for the multi-modal anomaly detection and localization tasks in the medical community. The code is available at https://github.com/Jingtao-Li-CVer/MMRAD.

Retinomorphic Object Detection in Asynchronous Visual Streams.

Retinomorphic Sensing

Adapting the Segment Anything Model for Multi-modal Retinal Anomaly Detection and Localization

Retina-Inspired Object Motion Segmentation for Event-Cameras

Event-Based Neuromorphic Vision for Autonomous Driving: A Paradigm Shift for Bio-Inspired Visual Sensing and Perception.

SODFormer: Streaming Object Detection with Transformer Using Events and Frames

CRSOT: Cross-Resolution Object Tracking using Unaligned Frame and Event Cameras

Real-time Tracking Based on Neuromrophic Vision

NeuSpike-Net: High Speed Video Reconstruction via Bio-inspired Neuromorphic Cameras

Accelerated Event-Based Feature Detection and Compression for Surveillance Video Systems

Neuromorphic Imaging With Joint Image Deblurring and Event Denoising

Neuromorphic Imaging with Super-Resolution

Modeling bio-inspired visual neural for detecting visual features of small- and wide-field moving targets synchronously from complex dynamic environments

Ultra-high Temporal Resolution Visual Reconstruction from a Fovea-like Spike Camera via Spiking Neuron Model

Event-based Robotic Grasping Detection with Neuromorphic Vision Sensor and Event-Stream Dataset

A Retinotopic Spiking Neural Network System for Accurate Recognition of Moving Objects Using NeuCube and Dynamic Vision Sensors

CIFAR10-DVS: an Event-Stream Dataset for Object Classification

Multi-modal Streaming 3D Object Detection

StreamYOLO: Real-time Object Detection for Streaming Perception