Abstract:Masked Image Modelling (MIM), a form of self-supervised learning, has garnered significant success in computer vision by improving image representations using unannotated data. Traditional MIMs typically employ a strategy of random sampling across the image. However, this random masking technique may not be ideally suited for medical imaging, which possesses distinct characteristics divergent from natural images. In medical imaging, particularly in pathology, disease-related features are often exceedingly sparse and localized, while the remaining regions appear normal and undifferentiated. Additionally, medical images frequently accompany reports, directly pinpointing pathological changes' location. Inspired by this, we propose Masked medical Image Modelling (MedIM), a novel approach, to our knowledge, the first research that employs radiological reports to guide the masking and restore the informative areas of images, encouraging the network to explore the stronger semantic representations from medical images. We introduce two mutual comprehensive masking strategies, knowledge-driven masking (KDM), and sentence-driven masking (SDM). KDM uses Medical Subject Headings (MeSH) words unique to radiology reports to identify symptom clues mapped to MeSH words (e.g., cardiac, edema, vascular, pulmonary) and guide the mask generation. Recognizing that radiological reports often comprise several sentences detailing varied findings, SDM integrates sentence-level information to identify key regions for masking. MedIM reconstructs images informed by this masking from the KDM and SDM modules, promoting a comprehensive and enriched medical image representation. Our extensive experiments on seven downstream tasks covering multi-label/class image classification, pneumothorax segmentation, and medical image-report analysis, demonstrate that MedIM with report-guided masking achieves competitive performance. Our method substantially outperforms ImageNet pre-training, MIM-based pre-training, and medical image-report pre-training counterparts. Codes are available at https://github.com/YtongXie/MedIM.

MIM-OOD: Generative Masked Image Modelling for Out-of-Distribution Detection in Medical Images

Achieving state-of-the-art performance in the Medical Out-of-Distribution (MOOD) challenge using plausible synthetic anomalies

MOODv2: Masked Image Modeling for Out-of-Distribution Detection

LADMIM: Logical Anomaly Detection with Masked Image Modeling in Discrete Latent Space

Rethinking Out-of-distribution (OOD) Detection: Masked Image Modeling is All You Need

Towards Reliable AI Model Deployments: Multiple Input Mixup for Out-of-Distribution Detection

Adversarial Masked Image Inpainting for Robust Detection of Mpox and Non-Mpox

Unsupervised out-of-distribution detection for safer robotically guided retinal microsurgery

Rethinking masked image modelling for medical image representation

A Unified Approach to Semi-Supervised Out-of-Distribution Detection

MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities

MIMRD: an Enhanced Masked Image Modeling Reverse Distillation for Anomaly Detection

Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection Capability

Unmasking the Chameleons: A Benchmark for Out-of-Distribution Detection in Medical Tabular Data

Multi-layer Aggregation as a key to feature-based OOD detection

Enhancing Reconstruction-Based Out-of-Distribution Detection in Brain MRI with Model and Metric Ensembles

Masked Image Modeling Advances 3D Medical Image Analysis

MOOD 2020: A public Benchmark for Out-of-Distribution Detection and Localization on medical Images

Unsupervised Anomaly Detection in Medical Images Using Masked Diffusion Model

DIAGNOSE: Avoiding Out-of-distribution Data using Submodular Information Measures

Redesigning Out-of-Distribution Detection on 3D Medical Images