Image completion using structure and texture GAN network.

Jingtao Guo,Yi Liu

DOI: https://doi.org/10.1016/j.neucom.2019.06.010

IF: 6

2019-01-01

Neurocomputing

Abstract:Recently, image completion using deep neural networks has become a hot research area. As structure (the underlying sketch) and texture (the pixel pattern mapped onto semantic structure) are two indispensable parts of images, completion algorithms must deal with them appropriately to produce realistic results. However, many current methods use the end-to-end (such as encode–decode) framework to repair an image, which do not pay special attention to texture and structure. Therefore, they often generate distorted structures and inconsistent textures. Moreover, they also do not allow a user to specify structure and texture information arbitrarily in the missing region. To this end, we propose a novel image completion method comprising a sketch completion network (sGAN) followed by a texture completion network (tGAN). sGAN focuses on repairing the sketch structures in the missing region of an image, and tGAN generates consistent texture information in the missing region based on the sketch output by sGAN and the surrounding incomplete image. By modeling the two parts separately in a deep neural network, not only the proposed method can successfully synthesize semantically valid and visually plausible contents in the missing region, but also it allows a user to manipulate the structure attributes freely in that region. Experimental results on the CelebA, Places2 and CelebA-HQ datasets demonstrate our method owns superior performance on challenging image completion examples and it provides far more user control flexibility than nearly all existing methods.