Abstract:Learning compact and meaningful latent space representations has been shown to be very useful in generative modeling tasks for visual data. One particular example is applying Vector Quantization (VQ) in variational autoencoders (VQ-VAEs, VQ-GANs, etc.), which has demonstrated state-of-the-art performance in many modern generative modeling applications. Quantizing the latent space has been justified by the assumption that the data themselves are inherently discrete in the latent space (like pixel values). In this paper, we propose an alternative representation of the latent space by relaxing the structural assumption than the VQ formulation. Specifically, we assume that the latent space can be approximated by a union of subspaces model corresponding to a dictionary-based representation under a sparsity constraint. The dictionary is learned/updated during the training process. We apply this approach to look at two models: Dictionary Learning Variational Autoencoders (DL-VAEs) and DL-VAEs with Generative Adversarial Networks (DL-GANs). We show empirically that our more latent space is more expressive and has leads to better representations than the VQ approach in terms of reconstruction quality at the expense of a small computational overhead for the latent space computation. Our results thus suggest that the true benefit of the VQ approach might not be from discretization of the latent space, but rather the lossy compression of the latent space. We confirm this hypothesis by showing that our sparse representations also address the codebook collapse issue as found common in VQ-family models.

Enforcing Sparsity on Latent Space for Robust and Explainable Representations

Learning Sparse Latent Representations for Generator Model

Efficient Probabilistic Latent Semantic Analysis with Sparsity Control

Learning Sparsity of Representations with Discrete Latent Variables

Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity

LASERS: LAtent Space Encoding for Representations with Sparsity for Generative Modeling

Nonsparse learning with latent variables

Inducing Sparse Coding and And-Or Grammar from Generator Network

SPARLING: Learning Latent Representations with Extremely Sparse Activations

Sparse Expansion and Neuronal Disentanglement

Neural Generators of Sparse Local Linear Models for Achieving both Accuracy and Interpretability

From Bayesian Sparsity to Gated Recurrent Nets

Sparse Activity and Sparse Connectivity in Supervised Learning

LatentExplainer: Explaining Latent Representations in Deep Generative Models with Multi-modal Foundation Models

Sparsity-aware generalization theory for deep neural networks

Encoding Domain Information with Sparse Priors for Inferring Explainable Latent Variables

Sparse encoding for more-interpretable feature-selecting representations in probabilistic matrix factorization

Associating Latent Representations with Cognitive Maps Via Hyperspherical Space for Neural Population Spikes.

Sparse word embeddings using l1 regularized online learning

Sparsity Winning Twice: Better Robust Generalization from More Efficient Training

SPINE: SParse Interpretable Neural Embeddings