Data Augmentation for Surgical Scene Segmentation with Anatomy-Aware Diffusion Models

Danush Kumar Venkatesh,Dominik Rivoir,Micha Pfeiffer,Fiona Kolbinger,Stefanie Speidel

2024-11-21

Abstract:In computer-assisted surgery, automatically recognizing anatomical organs is crucial for understanding the surgical scene and providing intraoperative assistance. While machine learning models can identify such structures, their deployment is hindered by the need for labeled, diverse surgical datasets with anatomical annotations. Labeling multiple classes (i.e., organs) in a surgical scene is time-intensive, requiring medical experts. Although synthetically generated images can enhance segmentation performance, maintaining both organ structure and texture during generation is challenging. We introduce a multi-stage approach using diffusion models to generate multi-class surgical datasets with annotations. Our framework improves anatomy awareness by training organ specific models with an inpainting objective guided by binary segmentation masks. The organs are generated with an inference pipeline using pre-trained ControlNet to maintain the organ structure. The synthetic multi-class datasets are constructed through an image composition step, ensuring structural and textural consistency. This versatile approach allows the generation of multi-class datasets from real binary datasets and simulated surgical masks. We thoroughly evaluate the generated datasets on image quality and downstream segmentation, achieving a $15\%$ improvement in segmentation scores when combined with real images. The code is available at <a class="link-external link-https" href="https://gitlab.com/nct_tso_public/muli-class-image-synthesis" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenge of automatically identifying anatomical organs in computer - assisted surgery. Specifically, although machine - learning models are able to recognize these structures, their application is limited by the need for large - scale, diverse surgical datasets with anatomical annotations. Annotating multiple categories (i.e., organs) is very time - consuming in the surgical scenario and requires the participation of medical experts. Moreover, although synthetically generated images can improve the segmentation performance, maintaining the consistency of organ structures and textures during the generation process remains a challenge. Therefore, this paper proposes a multi - stage method, using a diffusion model to generate multi - class surgical datasets with annotations to improve anatomical awareness and maintaining the organ structures through pre - trained ControlNet, thereby solving the above - mentioned problems.

Data Augmentation for Surgical Scene Segmentation with Anatomy-Aware Diffusion Models

AnatoMix: Anatomy-aware Data Augmentation for Multi-organ Segmentation

Image Synthesis with Class-Aware Semantic Diffusion Models for Surgical Scene Segmentation

Guided image generation for improved surgical image segmentation

SSIS-Seg: Simulation-Supervised Image Synthesis for Surgical Instrument Segmentation

Anatomically-Controllable Medical Image Generation with Segmentation-Guided Diffusion Models

SurgicaL-CD: Generating Surgical Images via Unpaired Image Translation with Latent Consistency Diffusion Models

Handling Geometric Domain Shifts in Semantic Segmentation of Surgical RGB and Hyperspectral Images

Data Augmentation in Class-Conditional Diffusion Model for Semi-Supervised Medical Image Segmentation

Artificial Intelligence Generated Data Augmentation for Abdominal Multi-Organ Segmentation

An Augmentation Strategy for Medical Image Processing Based on Statistical Shape Model and 3D Thin Plate Spline for Deep Learning

Generalizing Surgical Instruments Segmentation to Unseen Domains with One-to-Many Synthesis

Diverse Data Augmentation for Learning Image Segmentation with Cross-Modality Annotations.

SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Models

3D MRI Synthesis with Slice-Based Latent Diffusion Models: Improving Tumor Segmentation Tasks in Data-Scarce Regimes

One model to use them all: Training a segmentation model with complementary datasets

Synthesizing Images With Annotations for Medical Image Segmentation Using Diffusion Probabilistic Model

Using Diffusion Models to Generate Synthetic Labelled Data for Medical Image Segmentation

Semantic segmentation of surgical hyperspectral images under geometric domain shifts

Cascaded Diffusion Models for 2D and 3D Microscopy Image Synthesis to Enhance Cell Segmentation

Interpretability-guided Data Augmentation for Robust Segmentation in Multi-centre Colonoscopy Data