Abstract:Although the recent learning-based image and video coding techniques achieve rapid development, the signal fidelity-driven target in these methods leads to the divergence to a highly effective and efficient coding framework for both human and machine. In this paper, we aim to address the issue by making use of the power of generative models to bridge the gap between full fidelity (for human vision) and high discrimination (for machine vision). Therefore, relying on existing pretrained generative adversarial networks (GAN), we build a GAN inversion framework that projects the image into a low-dimensional natural image manifold. In this manifold, the feature is highly discriminative and also encodes the appearance information of the image, named as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">latent code</i> . Taking a variational bit-rate constraint with a hyperprior model to model/suppress the entropy of image manifold code, our method is capable of fulfilling the needs of both machine and human visions at very low bit-rates. To improve the visual quality of image reconstruction, we further propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multiple latent codes</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">scalable inversion</i> . The former gets several latent codes in the inversion, while the latter additionally compresses and transmits a shallow compact feature to support visual reconstruction. Experimental results demonstrate the superiority of our method in both human vision tasks, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> . image reconstruction, and machine vision tasks, including semantic parsing and attribute prediction.

Generative Adversarial Networks for Non-Negative Matrix Factorization in Temporal Psycho-Visual Modulation

SpatialGAN: Progressive Image Generation Based on Spatial Recursive Adversarial Expansion

Evaluation of Different Algorithms of Nonnegative Matrix Factorization in Temporal Psychovisual Modulation

Temporal conditional Wasserstein GANs for audio-visual affect-related ties

Progressive Deep Non-Negative Matrix Factorization Architecture with Graph Convolution-based Basis Image Reorganization

Factorization Algorithms for Temporal Psychovisual Modulation Display

Spatially-weighted Nonnegative Matrix Factorization with Application to Temporal Psychovisual Modulation

Spatially correlated nonnegative matrix factorization

Neighborhood Preserving Non-Negative Tensor Factorization For Image Representation

Graph Regularized Discriminative Non-Negative Matrix Factorization for Face Recognition

Spatially correlated nonnegative matrix factorization for image analysis

Temporal Information Guided Generative Adversarial Networks for Stimuli Image Reconstruction from Human Brain Activities

Reconstructing Rapid Natural Vision with fMRI-Conditional Video Generative Adversarial Network

Semi-supervised Convex Nonnegative Matrix Factorizations with Graph Regularized for Image Representation

Facial Image Compression via Neural Image Manifold Compression

Adversarially-Trained Nonnegative Matrix Factorization

Bilinear Models of Parts and Appearances in Generative Adversarial Networks

Multimodal Image-to-image Translation Between Domains with High Internal Variability

Parallel Vector Field Regularized Non-Negative Matrix Factorization For Image Representation

Robust Nonnegative Matrix Factorization with Discriminability for Image Representation

Ae-OT: a New Generative Model Based on Extended Semi-discrete Optimal Transport