Real-Time Image Inpainting using PatchMatch Based Two-Generator Adversarial Networks with Optimized Edge Loss Function.

Luchang Ding,Jing Zhang,Chang Wu,Chang Cai,Gengsheng Chen
DOI: https://doi.org/10.1109/ISCAS48785.2022.9937276
2022-01-01
Abstract:Image inpainting algorithms based on the deep learning are rapidly developed in the past few years, as they can synthesize textures and structures while maintaining semantic continuity. However, due to the limitations of existing loss functions and convolution, the generated images are usually blurry and have artificial textures. In this paper, a real-time image inpainting system using PatchMatch based two-generator adversarial network (PatchMatch-GAN), is proposed to improve the clarity of generated images. The first generator is committed to continuous semantic textures, and the second one focuses on the image sharpness. Parallel-dilated convolution is used to enlarge the receptive field of filters. For the first generator, a pre-trained VGG16 is used as an encoder to extract features. For the second generator, a DCGAN-like network is designed to generate images. And a feature replacement method is utilized to improve the resolution. The edge loss has also been proposed as a part of the loss function to emphasize the role of the shape of objects. Experiments are carried out on place2 and irregular mask datasets. Compared with GLGAN and CAGAN, in PSNR, the improvements of our system are 3.33% and 2.7%, respectively; in FID, the improvements of our system are 59.54% and 35.59%, respectively. Moreover, the average processing time of the whole system is 29.96 ms, which is 6.56 times faster than GLGAN, and 21.33 times faster than CAGAN. These show that our system can achieve better image inpainting results while meeting the requirements of real-time processing speed.
What problem does this paper attempt to address?