Abstract:Learning rich data representations from unlabeled data is a key challenge towards applying deep learning algorithms in downstream tasks. Several variants of variational autoencoders (VAEs) have been proposed to learn compact data representations by encoding high-dimensional data in a lower dimensional space. Two main classes of VAEs methods may be distinguished depending on the characteristics of the meta-priors that are enforced in the representation learning step. The first class of methods derives a continuous encoding by assuming a static prior distribution in the latent space. The second class of methods learns instead a discrete latent representation using vector quantization (VQ) along with a codebook. However, both classes of methods suffer from certain challenges, which may lead to suboptimal image reconstruction results. The first class suffers from posterior collapse, whereas the second class suffers from codebook collapse. To address these challenges, we introduce a new VAE variant, termed sparse coding-based VAE with learned ISTA (SC-VAE), which integrates sparse coding within variational autoencoder framework. The proposed method learns sparse data representations that consist of a linear combination of a small number of predetermined orthogonal atoms. The sparse coding problem is solved using a learnable version of the iterative shrinkage thresholding algorithm (ISTA). Experiments on two image datasets demonstrate that our model achieves improved image reconstruction results compared to state-of-the-art methods. Moreover, we demonstrate that the use of learned sparse code vectors allows us to perform downstream tasks like image generation and unsupervised image segmentation through clustering image patches.

Pixel VQ-VAEs for Improved Pixel Art Representation

DualVAE: Controlling Colours of Generated and Real Images

Attentive VQ-VAE

Generating Diverse High-Fidelity Images with VQ-VAE-2

An Art-Oriented Pixelation Method for Cartoon Images

Pyramid-VAE-GAN: Transferring Hierarchical Latent Variables for Image Inpainting

VQ-DRAW: A Sequential Discrete VAE

VQ-NeRF: Vector Quantization Enhances Implicit Neural Representations

Application of Variational AutoEncoder (VAE) Model and Image Processing Approaches in Game Design

Catch Missing Details: Image Reconstruction with Frequency Augmented Variational Autoencoder

Real-Time Image Semantic Retrieval Based on VQ

Generating High-Resolution 3D Faces and Bodies Using VQ-VAE-2 with PixelSNAIL Networks on 2D Representations

ViCE: Improving Dense Representation Learning by Superpixelization and Contrasting Cluster Assignment

Efficient-VQGAN: Towards High-Resolution Image Generation with Efficient Vision Transformers

VSA4VQA: Scaling a Vector Symbolic Architecture to Visual Question Answering on Natural Images

ESVAE: An Efficient Spiking Variational Autoencoder with Reparameterizable Poisson Spiking Sampling

VQMG: Hierarchical Vector Quantised and Multi-hops Graph Reasoning for Explicit Representation Learning

Hierarchical Vector-Quantized Variational Autoencoder and Vector Credibility Mechanism for High-Quality Image Inpainting

Neural Discrete Representation Learning

SC-VAE: Sparse Coding-based Variational Autoencoder with Learned ISTA

Pixel Embedding: Fully Quantized Convolutional Neural Network with Differentiable Lookup Table