Abstract:This paper introduces an innovative methodology for producing high-quality 3D lung CT images guided by textual information. While diffusion-based generative models are increasingly used in medical imaging, current state-of-the-art approaches are limited to low-resolution outputs and underutilize radiology reports' abundant information. The radiology reports can enhance the generation process by providing additional guidance and offering fine-grained control over the synthesis of images. Nevertheless, expanding text-guided generation to high-resolution 3D images poses significant memory and anatomical detail-preserving challenges. Addressing the memory issue, we introduce a hierarchical scheme that uses a modified UNet architecture. We start by synthesizing low-resolution images conditioned on the text, serving as a foundation for subsequent generators for complete volumetric data. To ensure the anatomical plausibility of the generated samples, we provide further guidance by generating vascular, airway, and lobular segmentation masks in conjunction with the CT images. The model demonstrates the capability to use textual input and segmentation tasks to generate synthesized images. Algorithmic comparative assessments and blind evaluations conducted by 10 board-certified radiologists indicate that our approach exhibits superior performance compared to the most advanced models based on GAN and diffusion techniques, especially in accurately retaining crucial anatomical features such as fissure lines and airways. This innovation introduces novel possibilities. This study focuses on two main objectives: (1) the development of a method for creating images based on textual prompts and anatomical components, and (2) the capability to generate new images conditioning on anatomical elements. The advancements in image generation can be applied to enhance numerous downstream tasks.

Generative Enhancement for 3D Medical Images

MedGen3D: A Deep Generative Framework for Paired 3D Image and Mask Generation

3D MedDiffusion: A 3D Medical Diffusion Model for Controllable and High-quality Medical Image Generation

Denoising diffusion probabilistic models for 3D medical image generation

Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis

Three-dimensional Bone Image Synthesis with Generative Adversarial Networks

Conditional Diffusion Models for Semantic 3D Brain MRI Synthesis

3D MRI Synthesis with Slice-Based Latent Diffusion Models: Improving Tumor Segmentation Tasks in Data-Scarce Regimes

Deep Generative Models for 3D Medical Image Synthesis

Advanced image generation for cancer using diffusion models

Inflating 2D Convolution Weights for Efficient Generation of 3D Medical Images

Enabling Competitive Performance of Medical Imaging with Diffusion Model-generated Images without Privacy Leakage

End-to-End 3D Liver CT Image Synthesis from Vasculature Using a Multi-Task Conditional Generative Adversarial Network

Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks

Synthetic Medical Images from Dual Generative Adversarial Networks

Anatomically-Controllable Medical Image Generation with Segmentation-Guided Diffusion Models

3D-StyleGAN: A Style-Based Generative Adversarial Network for Generative Modeling of Three-Dimensional Medical Images

MedSyn: Text-Guided Anatomy-Aware Synthesis of High-Fidelity 3-D CT Images

A New Chapter for Medical Image Generation: The Stable Diffusion Method

MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Images

Enhancing Medical Imaging with GANs Synthesizing Realistic Images from Limited Data