Abstract:Multimodal models trained on large natural image-text pair datasets have exhibited astounding abilities in generating high-quality images. Medical imaging data is fundamentally different to natural images, and the language used to succinctly capture relevant details in medical data uses a different, narrow but semantically rich, domain-specific vocabulary. Not surprisingly, multi-modal models trained on natural image-text pairs do not tend to generalize well to the medical domain. Developing generative imaging models faithfully representing medical concepts while providing compositional diversity could mitigate the existing paucity of high-quality, annotated medical imaging datasets. In this work, we develop a strategy to overcome the large natural-medical distributional shift by adapting a pre-trained latent diffusion model on a corpus of publicly available chest x-rays (CXR) and their corresponding radiology (text) reports. We investigate the model's ability to generate high-fidelity, diverse synthetic CXR conditioned on text prompts. We assess the model outputs quantitatively using image quality metrics, and evaluate image quality and text-image alignment by human domain experts. We present evidence that the resulting model (RoentGen) is able to create visually convincing, diverse synthetic CXR images, and that the output can be controlled to a new extent by using free-form text prompts including radiology-specific language. Fine-tuning this model on a fixed training set and using it as a data augmentation method, we measure a 5% improvement of a classifier trained jointly on synthetic and real images, and a 3% improvement when trained on a larger but purely synthetic training set. Finally, we observe that this fine-tuning distills in-domain knowledge in the text-encoder and can improve its representation capabilities of certain diseases like pneumothorax by 25%.

Cascaded Latent Diffusion Models for High-Resolution Chest X-ray Synthesis

Chest-Diffusion: A Light-Weight Text-to-Image Model for Report-to-CXR Generation

Exploring Foundation Models for Synthetic Medical Imaging: A Study on Chest X-Rays and Fine-Tuning Techniques

A vision–language foundation model for the generation of realistic chest X-ray images

RoentGen: Vision-Language Foundation Model for Chest X-ray Generation

XReal: Realistic Anatomy and Pathology-Aware X-ray Generation via Controllable Diffusion Model

Diff-CXR: Report-to-CXR generation through a disease-knowledge enhanced diffusion model

Synthetic chest X-ray images from text prompts

Towards Predicting Temporal Changes in a Patient's Chest X-ray Images based on Electronic Health Records

Generation of Anonymous Chest Radiographs Using Latent Diffusion Models for Training Thoracic Abnormality Classification Systems

Adapted generative latent diffusion models for accurate pathological analysis in chest X-ray images

ChEX: Interactive Localization and Region Description in Chest X-rays

MediSyn: Text-Guided Diffusion Models for Broad Medical 2D and 3D Image Synthesis

CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset

BS-Diff: Effective Bone Suppression Using Conditional Diffusion Models from Chest X-Ray Images

Multiscale Latent Diffusion Model for Enhanced Feature Extraction from Medical Images

Spot the fake lungs: Generating Synthetic Medical Images using Neural Diffusion Models

Advanced image generation for cancer using diffusion models

Chest X-ray Inpainting with Deep Generative Models