Abstract:Although the recent learning-based image and video coding techniques achieve rapid development, the signal fidelity-driven target in these methods leads to the divergence to a highly effective and efficient coding framework for both human and machine. In this paper, we aim to address the issue by making use of the power of generative models to bridge the gap between full fidelity (for human vision) and high discrimination (for machine vision). Therefore, relying on existing pretrained generative adversarial networks (GAN), we build a GAN inversion framework that projects the image into a low-dimensional natural image manifold. In this manifold, the feature is highly discriminative and also encodes the appearance information of the image, named as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">latent code</i> . Taking a variational bit-rate constraint with a hyperprior model to model/suppress the entropy of image manifold code, our method is capable of fulfilling the needs of both machine and human visions at very low bit-rates. To improve the visual quality of image reconstruction, we further propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multiple latent codes</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">scalable inversion</i> . The former gets several latent codes in the inversion, while the latter additionally compresses and transmits a shallow compact feature to support visual reconstruction. Experimental results demonstrate the superiority of our method in both human vision tasks, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> . image reconstruction, and machine vision tasks, including semantic parsing and attribute prediction.

Manifold Fitting with CycleGAN.

Deep Manifold Computing and Visualization

Deep Manifold Computing and Visualization Using Elastic Locally Isometric Smoothness

Deep Manifold Transformation for Nonlinear Dimensionality Reduction

MMCGAN: Generative Adversarial Network with Explicit Manifold Prior

Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections

Exploring the Manifold of Neural Networks Using Diffusion Geometry

ManifoldNet: A Deep Network Framework for Manifold-valued Data

Manifold-Aware CycleGAN for High-Resolution Structural-to-DTI Synthesis

Geometry-Aware Generative Autoencoders for Warped Riemannian Metric Learning and Generative Modeling on Data Manifolds

MR-GAN: Manifold Regularized Generative Adversarial Networks

Manifold-preserved GANs

Manifold regularization with GANs for semi-supervised learning

Deep Manifold Part 1: Anatomy of Neural Network Manifold

Genetic Programming for Manifold Learning: Preserving Local Topology

A Geometric Understanding of Deep Learning

Manifold Constraint Regularization for Remote Sensing Image Generation

Learning Pose Image Manifolds Using Geometry-Preserving GANs and Elasticae

Manifold GCN: Diffusion-based Convolutional Neural Network for Manifold-valued Graphs

Facial Image Compression via Neural Image Manifold Compression

3D GANs and Latent Space: A comprehensive survey