Noel C. F. Codella,Ying Jin,Shrey Jain,Yu Gu,Ho Hin Lee,Asma Ben Abacha,Alberto Santamaria-Pang,Will Guyman,Naiteek Sangani,Sheng Zhang,Hoifung Poon,Stephanie Hyland,Shruthi Bannur,Javier Alvarez-Valle,Xue Li,John Garrett,Alan McMillan,Gaurav Rajguru,Madhu Maddi,Nilesh Vijayrania,Rehaan Bhimai,Nick Mecklenburg,Rupal Jain,Daniel Holstein,Naveen Gaur,Vijay Aski,Jenq-Neng Hwang,Thomas Lin,Ivan Tarapov,Matthew Lungren,Mu Wei

Abstract:In this work, we present MedImageInsight, an open-source medical imaging embedding model. MedImageInsight is trained on medical images with associated text and labels across a diverse collection of domains, including X-Ray, CT, MRI, dermoscopy, OCT, fundus photography, ultrasound, histopathology, and mammography. Rigorous evaluations demonstrate MedImageInsight's ability to achieve state-of-the-art (SOTA) or human expert level performance across classification, image-image search, and fine-tuning tasks. Specifically, on public datasets, MedImageInsight achieves SOTA in CT 3D medical image retrieval, as well as SOTA in disease classification and search for chest X-ray, dermatology, and OCT imaging. Furthermore, MedImageInsight achieves human expert performance in bone age estimation (on both public and partner data), as well as AUC above 0.9 in most other domains. When paired with a text decoder, MedImageInsight achieves near SOTA level single image report findings generation with less than 10\% the parameters of other models. Compared to fine-tuning GPT-4o with only MIMIC-CXR data for the same task, MedImageInsight outperforms in clinical metrics, but underperforms on lexical metrics where GPT-4o sets a new SOTA. Importantly for regulatory purposes, MedImageInsight can generate ROC curves, adjust sensitivity and specificity based on clinical need, and provide evidence-based decision support through image-image search (which can also enable retrieval augmented generation). In an independent clinical evaluation of image-image search in chest X-ray, MedImageInsight outperformed every other publicly available foundation model evaluated by large margins (over 6 points AUC), and significantly outperformed other models in terms of AI fairness (across age and gender). We hope releasing MedImageInsight will help enhance collective progress in medical imaging AI research and development.

VisionCLIP: An Med-AIGC based Ethical Language-Image Foundation Model for Generalizable Retina Image Analysis

EyeCLIP: A visual-language foundation model for multi-modal ophthalmic image analysis

RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports

A foundation model for generalizable disease detection from retinal images

EyeFound: A Multimodal Generalist Foundation Model for Ophthalmic Imaging

A Foundation LAnguage-Image model of the Retina (FLAIR): Encoding expert knowledge in text supervision

VisionFM: a Multi-Modal Multi-Task Vision Foundation Model for Generalist Ophthalmic Artificial Intelligence

MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging

Common and Rare Fundus Diseases Identification Using Vision-Language Foundation Model with Knowledge of Over 400 Diseases

Vision–language foundation model for echocardiogram interpretation

They're All Doctors: Synthesizing Diverse Counterfactuals to Mitigate Associative Bias

Medical Vision-Language Pre-Training for Brain Abnormalities

RetiGen: A Framework for Generalized Retinal Diagnosis Using Multi-View Fundus Images

OphGLM: An ophthalmology large language-and-vision assistant

RemoteCLIP: A Vision Language Foundation Model for Remote Sensing

VISION-MAE: A Foundation Model for Medical Image Segmentation and Classification

VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced with Clinical Knowledge

BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs

Visual–language Foundation Models in Medicine

Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images