Abstract:Large models pre-trained on massive data have become a flourishing paradigm of artificial intelligence systems. Recent works, such as M6, CogView, WenLan 2.0, NÜWA, and ERNIE-ViLG, further extend this diagram to joint Vision Language Pre-training (VLP). For VLP, the two-stage architecture is a popular design, which includes the first stage learning an encoding function of data and the second stage learning a probabilistic model of encoded representation of data. Vector quantization (VQ) has usually engaged in the encoding function of image data for the first stage. VQ includes a data structure (codebook) and an algorithm (finding nearest quantization). The publicly available VQ models (e.g., VQGAN, VQVAE, VQVAE2) include a codebook whose size is assigned empirically (e.g., 1024, 4096, and 16,384) by their authors. If we want a smaller codebook for a lower computation load of the VQ process, or we want a larger codebook for better reconstruction quality, we have to retrain VQ models that consist of the down-sampling net, the codebook, and the up-sampling net. However, retraining VQ models is very expensive since these models, with billions of parameters, are trained on massive datasets. It motivates us to find an approach to resize the codebook of Vector quantization without retraining. In this paper, we leverage hyperbolic embeddings to enhance codebook vectors with the co-occurrence information and reorder the enhanced codebook by the Hilbert curve. Then we can resize the codebook of vector quantization for lower computation load or better reconstruction quality. Experimental results prove the efficiency and effectiveness of our approach when compared with competitive baselines. The code will be released to the public.

Codebook Classification Rearrangement Vector Quantization

Vector Quantization: a Review

VQ Image Coding Using Sub-Vector Techniques.

An Efficient Codebook Search Algorithm for Line Spectrum Frequency (LSF) Vector Quantization in Speech Codec

An efficient encoding algorithm for vector quantization based on subvector technique.

Codebook Sharing in Multi-Stage Vector Quantization

&Lt;title>image Coding Using Variable-Rate Classified Side-Match Vector Quantization</title>

Fast Nearest-Neighbor Codeword Search Algorithms for Vector Quantization

Image Coding Using SMVQ with Two-Level Block Classifier

Image Coding Based on Classified Side-Match Vector Quantization

A General Codebook Design Method for Vector Quantization

Fast Codebook Design Method for Image Vector Quantisation

VQ Codebook Design Using Modified K-means Algorithm with Feature Classification and Grouping Based Initialization

Edge and Contrast Classified K-means Algorithm for Image Vector Quantizer Design.

Resizing codebook of vector quantization without retraining

Equal-Average Equal-Variance Equal-Norm Nearest Neighbor Search Algorithm for Vector Quantization

Vector Quantization Based On Genetic Algorithm

Vector Quantization Based On Genetic Simulated Annealing

Structural Codebook Design And Address-Dependent Vector Quantization

Optimization Algorithm of Vector Quantization for Noise Channel

Image Compression Algorithms Based on Side-Match Vector Quantizer with Gradient-Based Classifiers