Noel C. F. Codella,Ying Jin,Shrey Jain,Yu Gu,Ho Hin Lee,Asma Ben Abacha,Alberto Santamaria-Pang,Will Guyman,Naiteek Sangani,Sheng Zhang,Hoifung Poon,Stephanie Hyland,Shruthi Bannur,Javier Alvarez-Valle,Xue Li,John Garrett,Alan McMillan,Gaurav Rajguru,Madhu Maddi,Nilesh Vijayrania,Rehaan Bhimai,Nick Mecklenburg,Rupal Jain,Daniel Holstein,Naveen Gaur,Vijay Aski,Jenq-Neng Hwang,Thomas Lin,Ivan Tarapov,Matthew Lungren,Mu Wei

Abstract:In this work, we present MedImageInsight, an open-source medical imaging embedding model. MedImageInsight is trained on medical images with associated text and labels across a diverse collection of domains, including X-Ray, CT, MRI, dermoscopy, OCT, fundus photography, ultrasound, histopathology, and mammography. Rigorous evaluations demonstrate MedImageInsight's ability to achieve state-of-the-art (SOTA) or human expert level performance across classification, image-image search, and fine-tuning tasks. Specifically, on public datasets, MedImageInsight achieves SOTA in CT 3D medical image retrieval, as well as SOTA in disease classification and search for chest X-ray, dermatology, and OCT imaging. Furthermore, MedImageInsight achieves human expert performance in bone age estimation (on both public and partner data), as well as AUC above 0.9 in most other domains. When paired with a text decoder, MedImageInsight achieves near SOTA level single image report findings generation with less than 10\% the parameters of other models. Compared to fine-tuning GPT-4o with only MIMIC-CXR data for the same task, MedImageInsight outperforms in clinical metrics, but underperforms on lexical metrics where GPT-4o sets a new SOTA. Importantly for regulatory purposes, MedImageInsight can generate ROC curves, adjust sensitivity and specificity based on clinical need, and provide evidence-based decision support through image-image search (which can also enable retrieval augmented generation). In an independent clinical evaluation of image-image search in chest X-ray, MedImageInsight outperformed every other publicly available foundation model evaluated by large margins (over 6 points AUC), and significantly outperformed other models in terms of AI fairness (across age and gender). We hope releasing MedImageInsight will help enhance collective progress in medical imaging AI research and development.

Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning

EVA-X: A Foundation Model for General Chest X-ray Analysis with Self-supervised Learning

Self-supervised Multi-modal Training from Uncurated Image and Reports Enables Zero-shot Oversight Artificial Intelligence in Radiology

Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks

Foundation Models in Radiology: What, How, When, Why and Why Not

A foundation model for generalizable disease diagnosis in chest X-ray images

SyntheX: Scaling Up Learning-based X-ray Image Analysis Through In Silico Experiments

Self-supervised multi-modal training from uncurated images and reports enables monitoring AI in radiology

Exploring Foundation Models for Synthetic Medical Imaging: A Study on Chest X-Rays and Fine-Tuning Techniques

A foundation model for generalized brain MRI analysis

Demographic Bias of Expert-Level Vision-Language Foundation Models in Medical Imaging

ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders

Pre-training on High Definition X-ray Images: An Experimental Study

Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data

MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging

AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation

BarlowTwins-CXR : Enhancing Chest X-Ray abnormality localization in heterogeneous data with cross-domain self-supervised learning

Synthetic data accelerates the development of generalizable learning-based algorithms for X-ray image analysis

Foundation model for cancer imaging biomarkers

AI-Driven Thoracic X-ray Diagnostics: Transformative Transfer Learning for Clinical Validation in Pulmonary Radiography

RAD-DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision