MS-GAN: multi-scale GAN with parallel class activation maps for image reconstruction

Jian Rao,Aihua Ke,Gang Liu,Yue Ming
DOI: https://doi.org/10.1007/s00371-022-02468-4
2022-01-01
Abstract:Recently, image reconstruction has been a research hotspot in the field of deep learning. For image reconstruction, generative adversarial networks (GANs) have obtained some remarkable results, but the existing methods based on GANs have not achieved satisfactory reconstructed results in quality. In order to improve the quality, we propose a more effective multi-scale GAN for image reconstruction and the proposed method is called MS-GAN. The generator of MS-GAN uses an improved U-net to capture the important details from the sparse inputs. In MS-GAN, the parallel class activation maps (P-CAMs) and spectral normalization (SN) are added to U-net. P-CAMs are composed of two parallel class activation maps (CAMs) and can specifically guide the generator to focus on the important details of the images for a more realistic visual effect. For the training process, MS-GAN consists of two phases: the generating phase and the refinement phase. The generating phase is to use binary sparse edges and color domains to generate the preliminary images. The refinement phase is to further improve the quality of the preliminary images. Experimental verifications are conducted on some datasets, which include edges2shoes, edges2handbags and Getchu. Experimental results show that our approach outperforms the existing state-of-the-art methods. The images reconstructed by MS-GAN is more photo-realistic in terms of visual effects.
What problem does this paper attempt to address?