Abstract:Vector quantization is a technique in machine learning that discretizes continuous representations into a set of discrete vectors. It is widely employed in tokenizing data representations for large language models, diffusion models, and other generative models. Despite its prevalence, the characteristics and behaviors of vector quantization in generative models remain largely underexplored. In this study, we investigate representation collapse in vector quantization - a critical degradation where codebook tokens or latent embeddings lose their discriminative power by converging to a limited subset of values. This collapse fundamentally compromises the model's ability to capture diverse data patterns. By leveraging both synthetic and real datasets, we identify the severity of each type of collapses and triggering conditions. Our analysis reveals that restricted initialization and limited encoder capacity result in tokens collapse and embeddings collapse. Building on these findings, we propose potential solutions aimed at mitigating each collapse. To the best of our knowledge, this is the first comprehensive study examining representation collapsing problems in vector quantization.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the representation collapse problem that occurs during the vector quantization (VQ) process, which seriously affects the quality and performance of the generative model. Specifically, the author focuses on two types of collapse phenomena: 1. **Tokens Collapse**: - Problem description: In the discrete codebook, a large number of tokens are concentrated on a few embedding vectors, resulting in an insufficient number of tokens assigned to other embedding vectors, thus losing the diversity of data. - Impact: This collapse will cause the generated results to be too single, lacking the necessary diversity and precision, and affecting the practicality and extensibility of the model. 2. **Embeddings Collapse**: - Problem description: Due to insufficient encoder parameters, different categories of input data are clustered after being processed by the encoder, which hinders the learning of discrete representations and ultimately leads to embedding collapse. - Impact: This collapse makes the model unable to learn meaningful discrete representations, resulting in distorted generated results and information loss. ### Specific manifestations - **Tokens Collapse**: As shown in Figure 1 (left), tokens are concentrated in the central peak of the embedding distribution instead of being evenly distributed among the peaks, resulting in some data patterns not being fully represented. - **Embeddings Collapse**: As shown in Figure 1 (right), most areas of the embedding space collapse into a limited representation, losing important information in the original data. ### Solutions To address these collapse problems, the author proposes the following solutions: 1. **For Tokens Collapse**: - **Pre - training + Fine - tuning Strategy**: First, use an auto - encoder without VQ for pre - training, and then use the pre - trained weights to initialize VQ - VAE for fine - tuning. This enables tokens to have better semantic discrimination, thereby reducing the occurrence of token collapse. - Experimental verification: Through experiments on synthetic datasets and the CIFAR - 10 dataset, the effectiveness of this method has been proven. 2. **For Embeddings Collapse**: - **Increase Encoder Parameters**: By increasing the number of encoder parameters, improve its perception ability, thereby avoiding embedding collapse. - Experimental verification: By adjusting the hidden layer size of the encoder, it has been verified that insufficient encoder capacity will indeed lead to embedding collapse, and increasing parameters can effectively alleviate this problem. ### Summary This paper systematically studies the representation collapse problem in vector quantization, reveals that random initialization and insufficient encoder capacity are the main reasons for these two types of collapse, and proposes corresponding solutions. These findings provide an important theoretical basis and practical guidance for improving VQ technology and its application in generative models.

Representation Collapsing Problems in Vector Quantization

Addressing Representation Collapse in Vector Quantized Models with One Linear Layer

Distributed Vector Quantization Based On Kullback-Leibler Divergence

Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective

Restructuring Vector Quantization with the Rotation Trick

HyperVQ: MLR-based Vector Quantization in Hyperbolic Space

Distributed Vector Quantization over Sensor Network

Quantifying the Variability Collapse of Neural Networks

Resizing codebook of vector quantization without retraining

Scalar Quantization as Sparse Least Square Optimization

On the Representation Collapse of Sparse Mixture of Experts

VecQ: Minimal Loss DNN Model Compression With Vectorized Weight Quantization

The Prevalence of Neural Collapse in Neural Multivariate Regression

Soft Convex Quantization: Revisiting Vector Quantization with Convex Optimization

Exploring Neural Networks Quantization via Layer-Wise Quantization Analysis

Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders

Towards understanding neural collapse in supervised contrastive learning with the information bottleneck method

AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations

Ternary Quantization: A Survey

Vector Quantized Wasserstein Auto-Encoder

Quantization without Tears