Abstract:Vector-quantized networks (VQNs) have exhibited remarkable performance across various tasks, yet they are prone to training instability, which complicates the training process due to the necessity for techniques such as subtle initialization and model distillation. In this study, we identify the local minima issue as the primary cause of this instability. To address this, we integrate an optimal transport method in place of the nearest neighbor search to achieve a more globally informed assignment. We introduce OptVQ, a novel vector quantization method that employs the Sinkhorn algorithm to optimize the optimal transport problem, thereby enhancing the stability and efficiency of the training process. To mitigate the influence of diverse data distributions on the Sinkhorn algorithm, we implement a straightforward yet effective normalization strategy. Our comprehensive experiments on image reconstruction tasks demonstrate that OptVQ achieves 100% codebook utilization and surpasses current state-of-the-art VQNs in reconstruction quality.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the instability problems encountered during the training of vector - quantified networks (VQNs), especially the "index collapse" phenomenon caused by local minima. Specifically: 1. **Training Instability**: - VQNs are prone to fall into local minima during training, which complicates the training process and requires some special technical means to stabilize the training, such as fine - grained initialization and model distillation. - The paper points out that this instability mainly stems from the traditional nearest - neighbor search strategy, which is a greedy quantization method and is likely to result in only a few codebooks being used while most codebooks are not fully utilized. 2. **Local Minima Problem**: - Nearest - neighbor search is carried out based on local information, which may cause feature points to be trapped in certain specific Voronoi cells and unable to effectively utilize global information. - This local search strategy makes VQNs prone to fall into local optimal solutions, thus affecting the performance and stability of the model. To solve the above problems, the paper proposes a new vector quantization method - OptVQ. This method transforms the vector quantization problem into a global optimization problem by introducing the optimal transport theory, thereby avoiding the influence of local minima and improving the stability and efficiency of training. ### Core Improvements of OptVQ - **Optimal Transport Perspective**: View vector quantization as an optimal transport problem and use the Sinkhorn algorithm to optimize the transport problem to ensure the effective utilization of global information. - **Improve Codebook Utilization Rate**: Through the global optimization strategy, OptVQ can achieve 100% codebook utilization rate and avoid the "index collapse" phenomenon. - **Enhance Reconstruction Quality**: Experimental results show that OptVQ surpasses the existing state - of - the - art methods in image reconstruction tasks, not only improving the reconstruction quality but also enhancing the training stability. Through these improvements, OptVQ not only solves the instability problems in VQNs training but also demonstrates excellent performance on multiple datasets.

Preventing Local Pitfalls in Vector Quantization via Optimal Transport

Distributed Vector Quantization Based On Kullback-Leibler Divergence

Optimizing Quantized Neural Networks in a Weak Curvature Manifold

Distributed Vector Quantization over Sensor Network

Optimized Residual Vector Quantization for Efficient Approximate Nearest Neighbor Search

Equal-Average Equal-Variance Equal-Norm Nearest Neighbor Search Algorithm for Vector Quantization

DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformers

Soft Convex Quantization: Revisiting Vector Quantization with Convex Optimization

LibVQ: A Toolkit for Optimizing Vector Quantization and Efficient Neural Retrieval.

Fast Norm-Ordered Nearest-Neighbor Codeword Search Algorithms For Image Vector Quantization

VecQ: Minimal Loss DNN Model Compression With Vectorized Weight Quantization

NVTC: Nonlinear Vector Transform Coding

Regularized Vector Quantization for Tokenized Image Synthesis

Restructuring Vector Quantization with the Rotation Trick

Vector Quantization Using Survival Cauchy-Schwartz Divergence

Vector Quantization: a Review

Improved Residual Vector Quantization for High-dimensional Approximate Nearest Neighbor Search

Addressing Representation Collapse in Vector Quantized Models with One Linear Layer

ADFQ-ViT: Activation-Distribution-Friendly Post-Training Quantization for Vision Transformers

A Study of Methods for Improving Learning Vector Quantization

Learning Optimal Lattice Vector Quantizers for End-to-end Neural Image Compression