Abstract:Endoscopy is a routine imaging technique used for both diagnosis and minimally invasive surgical treatment. Artifacts such as motion blur, bubbles, specular reflections, floating objects and pixel saturation impede the visual interpretation and the automated analysis of endoscopy videos. Given the widespread use of endoscopy in different clinical applications, robust and reliable identification of such artifacts and the automated restoration of corrupted video frames is a fundamental medical imaging problem. Existing state-of-the-art methods only deal with the detection and restoration of selected artifacts. However, typically endoscopy videos contain numerous artifacts which motivates to establish a comprehensive solution.In this paper, a fully automatic framework is proposed that can: 1) detect and classify six different artifacts, 2) segment artifact instances that have indefinable shapes, 3) provide a quality score for each frame, and 4) restore partially corrupted frames. To detect and classify different artifacts, the proposed framework exploits fast, multi-scale and single stage convolution neural network detector. In addition, we use an encoder-decoder model for pixel-wise segmentation of irregular shaped artifacts. A quality score is introduced to assess video frame quality and to predict image restoration success. Generative adversarial networks with carefully chosen regularization and training strategies for discriminator-generator networks are finally used to restore corrupted frames.The detector yields the highest mean average precision (mAP) of 45.7 and 34.7, respectively for 25% and 50% IoU thresholds, and the lowest computational time of 88 ms allowing for near real-time processing. The restoration models for blind deblurring, saturation correction and inpainting demonstrate significant improvements over previous methods. On a set of 10 test videos, an average of 68.7% of video frames successfully passed the quality score (<math>≥0.9</math>) after applying the proposed restoration framework thereby retaining 25% more frames compared to the raw videos. The importance of artifacts detection and their restoration on improved robustness of image analysis methods is also demonstrated in this work.

Hierarchical Candidate Recursive Network for Highlight Restoration in Endoscopic Videos

Highlight Removal for Endoscopic Images based on Accelerated Adaptive Non-Convex RPCA Decomposition

SIHRNet: a fully convolutional network for single image highlight removal with a real-world dataset

Specular highlight removal for endoscopic images using partial attention network

Detection of Image Artifacts Using Improved Cascade Region-Based CNN for Quality Assessment of Endoscopic Images

Specular Highlight Detection Based on Color Distribution for Endoscopic Images

Enhancing endoscopic scene reconstruction with color-aware inverse rendering through neural SDF and radiance fields

Highlight Removal Network Based on an Improved Dichromatic Reflection Model.

Specular Reflections Removal for Endoscopic Image Sequences With Adaptive-RPCA Decomposition

Dual-Hybrid Attention Network for Specular Highlight Removal

Effective High-to-low-level Feature Aggregation Network for Endoscopic Image Classification

Pixel-wise Contrastive Learning for Multi-class Instrument Segmentation in Endoscopic Robotic Surgery Videos Using Dataset-wide Sample Queues

A deep learning framework for quality assessment and restoration in video endoscopy

CB-HRNet: A Class-Balanced High-Resolution Network for the evaluation of endoscopic activity in patients with ulcerative colitis

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its Effect on Image Correspondence

Specular Reflections Detection and Removal for Endoscopic Images Based on Brightness Classification

Noise reduction deep CNN-based retinal fundus image enhancement using recursive histogram

Retinex theory-based nonlinear luminance enhancement and denoising for low-light endoscopic images

Self-supervised neural network-based endoscopic monocular 3D reconstruction method

Semi-TSGAN: Semi-Supervised Learning for Highlight Removal Based on Teacher-Student Generative Adversarial Network