Reusing Discriminators for Encoding: Towards Unsupervised Image-to-Image Translation

Runfa Chen,Wenbing Huang,Binghui Huang,Fuchun Sun,Bin Fang

DOI: https://doi.org/10.48550/arXiv.2003.00273

2020-03-28

Abstract:Unsupervised image-to-image translation is a central task in computer vision. Current translation frameworks will abandon the discriminator once the training process is completed. This paper contends a novel role of the discriminator by reusing it for encoding the images of the target domain. The proposed architecture, termed as NICE-GAN, exhibits two advantageous patterns over previous approaches: First, it is more compact since no independent encoding component is required; Second, this plug-in encoder is directly trained by the adversary loss, making it more informative and trained more effectively if a multi-scale discriminator is applied. The main issue in NICE-GAN is the coupling of translation with discrimination along the encoder, which could incur training inconsistency when we play the min-max game via GAN. To tackle this issue, we develop a decoupled training strategy by which the encoder is only trained when maximizing the adversary loss while keeping frozen otherwise. Extensive experiments on four popular benchmarks demonstrate the superior performance of NICE-GAN over state-of-the-art methods in terms of FID, KID, and also human preference. Comprehensive ablation studies are also carried out to isolate the validity of each proposed component. Our codes are available at <a class="link-external link-https" href="https://github.com/alpc91/NICE-GAN-pytorch" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to use the discriminator more efficiently in unsupervised image - to - image translation tasks to construct a more compact and effective model architecture. Specifically, the authors propose a new method named NICE - GAN, which re - uses part of the discriminator as an encoder for image encoding in the target domain. This innovation not only reduces the number of model components and improves the compactness of the model, but also improves the effectiveness and training efficiency of the encoder by directly using adversarial loss to train the encoder. In addition, in order to overcome the training instability problem caused by the overlap between the encoder and the discriminator, the authors develop a decoupled training strategy, that is, only train the encoder when maximizing the adversarial loss, and keep the encoder unchanged in other cases. The experimental results on multiple benchmark datasets show that NICE - GAN is superior to the existing state - of - the - art methods in image translation quality, and has fewer model parameters and lower computational cost.

The main contributions of the paper include:
1. For the first time, it is proposed to re - use the discriminator for encoding in unsupervised image - to - image translation tasks, and a more compact and effective model architecture - NICE - GAN is constructed.
2. A simple decoupled training strategy is developed for the training instability problem that may be caused by re - using the discriminator.
3. Through extensive experimental evaluations, it is proved that the proposed method is superior to various state - of - the - art methods on multiple popular benchmark datasets. At the same time, an exhaustive ablation study is carried out to verify the effectiveness of each proposed component.