Abstract:Large models pre-trained on massive data have become a flourishing paradigm of artificial intelligence systems. Recent works, such as M6, CogView, WenLan 2.0, NÜWA, and ERNIE-ViLG, further extend this diagram to joint Vision Language Pre-training (VLP). For VLP, the two-stage architecture is a popular design, which includes the first stage learning an encoding function of data and the second stage learning a probabilistic model of encoded representation of data. Vector quantization (VQ) has usually engaged in the encoding function of image data for the first stage. VQ includes a data structure (codebook) and an algorithm (finding nearest quantization). The publicly available VQ models (e.g., VQGAN, VQVAE, VQVAE2) include a codebook whose size is assigned empirically (e.g., 1024, 4096, and 16,384) by their authors. If we want a smaller codebook for a lower computation load of the VQ process, or we want a larger codebook for better reconstruction quality, we have to retrain VQ models that consist of the down-sampling net, the codebook, and the up-sampling net. However, retraining VQ models is very expensive since these models, with billions of parameters, are trained on massive datasets. It motivates us to find an approach to resize the codebook of Vector quantization without retraining. In this paper, we leverage hyperbolic embeddings to enhance codebook vectors with the co-occurrence information and reorder the enhanced codebook by the Hilbert curve. Then we can resize the codebook of vector quantization for lower computation load or better reconstruction quality. Experimental results prove the efficiency and effectiveness of our approach when compared with competitive baselines. The code will be released to the public.

Codebook Sharing in Multi-Stage Vector Quantization

Mode Based Multi-Stage Vector Quantization with Shared Codebooks

Multi-Stage Codebook Optimization for LSF Quantization with Inter-Stage Prediction

VQ Image Coding Using Sub-Vector Techniques.

Fast Codebook Design Method for Image Vector Quantisation

Vector Quantization Based On Genetic Simulated Annealing

Joint Optimization Algorithm of Codebook Size in Multi-Parameter, Multi-Mode Quantization

Resizing codebook of vector quantization without retraining

An efficient encoding algorithm for vector quantization based on subvector technique.

VQ Codebook Design Using Modified K-means Algorithm with Feature Classification and Grouping Based Initialization

Structural Codebook Design And Address-Dependent Vector Quantization

Dynamic Extended Codebook Based Vector Quantization Scheme for Mesh Geometry Compression

Image Coding Using SMVQ with Two-Level Block Classifier

Vector Quantization Based On Genetic Algorithm

Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillation

High Efficiency MSVQ for Prediction Linear Spectrum Frequency Parameters with Inter-Frame and Inter-Stage Prediction

Joint optimization algorithm of multi-parameter codebook size based on superframe mode

&Lt;title>image Coding Using Variable-Rate Classified Side-Match Vector Quantization</title>

LG-VQ: Language-Guided Codebook Learning

A Fast Image Coding Algorithm Using Variable-Rate Mean-Match Correlation Vector Quantization

Dynamically restricted codebook-based vector quantisation scheme for mesh geometry compression