Abstract:Inpainting involves filling in missing pixels or areas in an image, a crucial technique employed in Mixed Reality environments for various applications, particularly in Diminished Reality (DR) where content is removed from a user's visual environment. Existing methods rely on digital replacement techniques which necessitate multiple cameras and incur high costs. AR devices and smartphones use ToF depth sensors to capture scene depth maps aligned with RGB images. Despite speed and affordability, ToF cameras create imperfect depth maps with missing pixels. To address the above challenges, we propose Hierarchical Inpainting GAN (HI-GAN), a novel approach comprising three GANs in a hierarchical fashion for RGBD inpainting. EdgeGAN and LabelGAN inpaint masked edge and segmentation label images respectively, while CombinedRGBD-GAN combines their latent representation outputs and performs RGB and Depth inpainting. Edge images and particularly segmentation label images as auxiliary inputs significantly enhance inpainting performance by complementary context and hierarchical optimization. We believe we make the first attempt to incorporate label images into inpainting process.Unlike previous approaches requiring multiple sequential models and separate outputs, our work operates in an end-to-end manner, training all three models simultaneously and hierarchically. Specifically, EdgeGAN and LabelGAN are first optimized separately and further optimized inside CombinedRGBD-GAN to enhance inpainting quality. Experiments demonstrate that HI-GAN works seamlessly and achieves overall superior performance compared with existing approaches.

Stereo Image Inpainting for UGVs Based on a Multistage Feature Alignment and Refinement Network

A Novel Stereo Image Self-Inpainting Network for Autonomous Robots

UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation

Virtual View Generation Based on 3D-Dense-attentive GAN Networks

Image Inpainting Based on Interactive Separation Network and Progressive Reconstruction Algorithm

Face Image Inpainting Based on Generative Adversarial Network

RSINet: Inpainting Remotely Sensed Images Using Triple GAN Framework

A Progressive Image Inpainting Algorithm with a Mask Auto-update Branch

High-Resolution Image Inpainting Through Multiple Deep Networks

Panoramic Image Inpainting With Gated Convolution And Contextual Reconstruction Loss

Cnn-Based Stereoscopic Image Inpainting

DGCA: high resolution image inpainting via DR-GAN and contextual attention

Image inpainting method based on AU-GAN

UGNet: Uncertainty aware geometry enhanced networks for stereo matching

Attentional coarse-and-fine generative adversarial networks for image inpainting

High-Resolution Image Inpainting Based On Multi-Scale Neural Network

A three-stage GAN model based on edge and color prediction for image outpainting

AGG: Attention-Based Gated Convolutional GAN with Prior Guidance for Image Inpainting

Distillation-guided Image Inpainting

Coarse-to-fine Task-driven Inpainting for Geoscience Images

HI-GAN: Hierarchical Inpainting GAN with Auxiliary Inputs for Combined RGB and Depth Inpainting