Abstract:Medical image analysis has significantly benefited from advancements in deep learning, particularly in the application of Generative Adversarial Networks (GANs) for generating realistic and diverse images that can augment training datasets. However, the effectiveness of such approaches is often limited by the amount of available data in clinical settings. Additionally, the common GAN-based approach is to generate entire image volumes, rather than solely the region of interest (ROI). Research on deep learning-based brain tumor classification using MRI has shown that it is easier to classify the tumor ROIs compared to the entire image volumes. In this work, we present a novel framework that uses vector-quantization GAN and a transformer incorporating masked token modeling to generate high-resolution and diverse 3D brain tumor ROIs that can be directly used as augmented data for the classification of brain tumor ROI. We apply our method to two imbalanced datasets where we augment the minority class: (1) the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2019 dataset to generate new low-grade glioma (LGG) ROIs to balance with high-grade glioma (HGG) class; (2) the internal pediatric LGG (pLGG) dataset tumor ROIs with BRAF V600E Mutation genetic marker to balance with BRAF Fusion genetic marker class. We show that the proposed method outperforms various baseline models in both qualitative and quantitative measurements. The generated data was used to balance the data in the brain tumor types classification task. Using the augmented data, our approach surpasses baseline models by 6.4% in AUC on the BraTS 2019 dataset and 4.3% in AUC on our internal pLGG dataset. The results indicate the generated tumor ROIs can effectively address the imbalanced data problem. Our proposed method has the potential to facilitate an accurate diagnosis of rare brain tumors using MRI scans.

Advancing Brain Tumor Inpainting with Generative Models

Fast and Lightweight Network Improves Serial Brain Section Stitching

Denoising Diffusion Models for Inpainting of Healthy Brain Tissue

Multitask Brain Tumor Inpainting with Diffusion Models: A Methodological Report

Ensemble Learning and 3D Pix2Pix for Comprehensive Brain Tumor Analysis in Multimodal MRI

Denoising Diffusion Models for 3D Healthy Brain Tissue Inpainting

Brain MR Atlas Construction Using Symmetric Deep Neural Inpainting

Deep learning-Based 3D inpainting of brain MR images

Symmetric-Constrained Irregular Structure Inpainting for Brain MRI Registration with Tumor Pathology

sTBI-GAN: An adversarial learning approach for data synthesis on traumatic brain segmentation

The Brain Tumor Segmentation (BraTS) Challenge: Local Synthesis of Healthy Brain Tissue via Inpainting

Generating 3D Brain Tumor Regions in MRI using Vector-Quantization Generative Adversarial Networks

Unsupervised Region-based Anomaly Detection in Brain MRI with Adversarial Image Inpainting

Generative Adversarial Network Using Multi-modal Guidance for Ultrasound Images Inpainting.

Brain Tumor Synthetic Segmentation in 3D Multimodal MRI Scans

A 3D generative model of pathological multi-modal MR images and segmentations

TBI-GAN: An Adversarial Learning Approach for Data Synthesis on Traumatic Brain Segmentation

Generative Learning-Based Lightweight MRI Brain Tumor Segmentation with Missing Modalities

MRI restoration using edge-guided adversarial learning

Inpainting Pathology in Lumbar Spine MRI with Latent Diffusion