Abstract:Large models pre-trained on massive data have become a flourishing paradigm of artificial intelligence systems. Recent works, such as M6, CogView, WenLan 2.0, NÜWA, and ERNIE-ViLG, further extend this diagram to joint Vision Language Pre-training (VLP). For VLP, the two-stage architecture is a popular design, which includes the first stage learning an encoding function of data and the second stage learning a probabilistic model of encoded representation of data. Vector quantization (VQ) has usually engaged in the encoding function of image data for the first stage. VQ includes a data structure (codebook) and an algorithm (finding nearest quantization). The publicly available VQ models (e.g., VQGAN, VQVAE, VQVAE2) include a codebook whose size is assigned empirically (e.g., 1024, 4096, and 16,384) by their authors. If we want a smaller codebook for a lower computation load of the VQ process, or we want a larger codebook for better reconstruction quality, we have to retrain VQ models that consist of the down-sampling net, the codebook, and the up-sampling net. However, retraining VQ models is very expensive since these models, with billions of parameters, are trained on massive datasets. It motivates us to find an approach to resize the codebook of Vector quantization without retraining. In this paper, we leverage hyperbolic embeddings to enhance codebook vectors with the co-occurrence information and reorder the enhanced codebook by the Hilbert curve. Then we can resize the codebook of vector quantization for lower computation load or better reconstruction quality. Experimental results prove the efficiency and effectiveness of our approach when compared with competitive baselines. The code will be released to the public.

Joint Optimization Algorithm of Codebook Size in Multi-Parameter, Multi-Mode Quantization

Joint optimization algorithm of multi-parameter codebook size based on superframe mode

Multi-Stage Codebook Optimization for LSF Quantization with Inter-Stage Prediction

Mode Based Multi-Stage Vector Quantization with Shared Codebooks

Codebook Sharing in Multi-Stage Vector Quantization

Edge and Contrast Classified K-means Algorithm for Image Vector Quantizer Design.

Fast Codebook Design Method for Image Vector Quantisation

Revisiting Multi-Codebook Quantization

An efficient encoding algorithm for vector quantization based on subvector technique.

VQ Codebook Design Using Modified K-means Algorithm with Feature Classification and Grouping Based Initialization

A Method Of Optimizing Codebook Based On Codeword Use Frequency

Resizing codebook of vector quantization without retraining

Robust joint source channel coding algorithm for speech coding

Optimization Algorithm of Vector Quantization for Noise Channel

A New Channel Optimized Vector Quantizer Based on the Partial Distortion Theorem

Joint Source-Channel Optimization of Vector Quantization with Polar Codes

Improved K-means Algorithm Using Initialization Technique Based on Edge-Mean Grid for Image Vector Quantizer Design.

Edge Classification Based Initialization Technique for Image Vector Quantizer Design.

Optimized Bit Allocation Scheme For Joint Source And Channel Coding

Pitch and Band-Pass Voicing Strengths Coefficients Joint Quantization Algorithm

Structural Codebook Design And Address-Dependent Vector Quantization