Abstract:Deep networks are increasingly being applied to problems involving image synthesis, e.g., generating images from textual descriptions and reconstructing an input image from a compact representation. Supervised training of image-synthesis networks typically uses a pixel-wise loss (PL) to indicate the mismatch between a generated image and its corresponding target image. We propose instead to use a loss function that is better calibrated to human perceptual judgments of image quality: the multiscale structural-similarity score (MS-SSIM). Because MS-SSIM is differentiable, it is easily incorporated into gradient-descent learning. We compare the consequences of using MS-SSIM versus PL loss on training deterministic and stochastic autoencoders. For three different architectures, we collected human judgments of the quality of image reconstructions. Observers reliably prefer images synthesized by MS-SSIM-optimized models over those synthesized by PL-optimized models, for two distinct PL measures ($\ell_1$ and $\ell_2$ distances). We also explore the effect of training objective on image encoding and analyze conditions under which perceptually-optimized representations yield better performance on image classification. Finally, we demonstrate the superiority of perceptually-optimized networks for super-resolution imaging. Just as computer vision has advanced through the use of convolutional architectures that mimic the structure of the mammalian visual system, we argue that significant additional advances can be made in modeling images through the use of training objectives that are well aligned to characteristics of human perception.

A Sliced Wasserstein Loss for Neural Texture Synthesis

Long Range Constraints for Neural Texture Synthesis Using Sliced Wasserstein Loss

Incorporating Multiscale Contextual Loss for Image Style Transfer

Multiview Textured Mesh Recovery by Differentiable Rendering

Texture Synthesis with Recurrent Variational Auto-Encoder

Texture Representation via Analysis and Synthesis with Generative Adversarial Networks

The Right Losses for the Right Gains: Improving the Semantic Consistency of Deep Text-to-Image Generation with Distribution-Sensitive Losses

Projected Distribution Loss for Image Enhancement

VCD-Texture: Variance Alignment based 3D-2D Co-Denoising for Text-Guided Texturing

Wasserstein Loss for Semantic Editing in the Latent Space of GANs

Texture Networks: Feed-forward Synthesis of Textures and Stylized Images

Convergence of SGD for Training Neural Networks with Sliced Wasserstein Losses

GramGAN: Deep 3D Texture Synthesis From 2D Exemplars

Efficient Texture Parameterization Driven by Perceptual‐Loss‐on‐Screen

Contrastive Feature Loss for Image Prediction

Generalized Rectifier Wavelet Covariance Models For Texture Synthesis

TextureWGAN: Texture Preserving WGAN with MLE Regularizer for Inverse Problems

Multi-Scale Texture Loss for CT denoising with GANs

Learning to Generate Images with Perceptual Similarity Metrics

Evaluating Lossy Compression Rates of Deep Generative Models

Compositional Neural Textures