Abstract:The lack of large datasets and high-quality annotated data often limits the development of accurate and robust machine-learning models within the medical and surgical domains. In the machine learning community, generative models have recently demonstrated that it is possible to produce novel and diverse synthetic images that closely resemble reality while controlling their content with various types of annotations. However, generative models have not been yet fully explored in the surgical domain, partially due to the lack of large datasets and due to specific challenges present in the surgical domain such as the large anatomical diversity. We propose Surgery-GAN, a novel generative model that produces synthetic images from segmentation maps. Our architecture produces surgical images with improved quality when compared to early generative models thanks to the combination of channel- and pixel-level normalization layers that boost image quality while granting adherence to the input segmentation map. While state-of-the-art generative models often generate overfitted images, lacking diversity, or containing unrealistic artefacts such as cartooning; experiments demonstrate that Surgery-GAN is able to generate novel, realistic, and diverse surgical images in three different surgical datasets: cholecystectomy, partial nephrectomy, and radical prostatectomy. In addition, we investigate whether the use of synthetic images together with real ones can be used to improve the performance of other machine-learning models. Specifically, we use Surgery-GAN to generate large synthetic datasets which we then use to train five different segmentation models. Results demonstrate that using our synthetic images always improves the mean segmentation performance with respect to only using real images. For example, when considering radical prostatectomy, we can boost the mean segmentation performance by up to 5.43%. More interestingly, experimental results indicate that the performance improvement is larger in the set of classes that are under-represented in the training sets, where the performance boost of specific classes reaches up to 61.6%.

Cross-Domain Conditional Generative Adversarial Networks for Stereoscopic Hyperrealism in Surgical Training

Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries

Guided image generation for improved surgical image segmentation

Using Conditional Generative Adversarial Networks to Reduce the Effects of Latency in Robotic Telesurgery

Creating Realistic Anterior Segment Optical Coherence Tomography Images using Generative Adversarial Networks

Augmentation of Virtual Endoscopic Images with Intra-operative Data using Content-Nets

SurgeoNet: Realtime 3D Pose Estimation of Articulated Surgical Instruments from Stereo Images using a Synthetically-trained Network

Surgical scene generation and adversarial networks for physics-based iOCT synthesis

Self-Supervised Generative Adversarial Network for Depth Estimation in Laparoscopic Images

Realistic Surgical Image Dataset Generation Based On 3D Gaussian Splatting

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

SurgicaL-CD: Generating Surgical Images via Unpaired Image Translation with Latent Consistency Diffusion Models

Local Style Preservation in Improved GAN-Driven Synthetic Image Generation for Endoscopic Tool Segmentation

Self-Supervised Siamese Learning on Stereo Image Pairs for Depth Estimation in Robotic Surgery

Stereo Video Reconstruction Without Explicit Depth Maps for Endoscopic Surgery

LC-GAN: Image-to-image Translation Based on Generative Adversarial Network for Endoscopic Images

Self-supervised Monocular Depth Estimation in Gastroendoscopy Using GAN-augmented Images

Towards Augmented Reality-based Suturing in Monocular Laparoscopic Training

Monocular pose estimation of articulated surgical instruments in open surgery