Abstract:Many common types of data can be represented as functions that map coordinates to signal values, such as pixel locations to RGB values in the case of an image. Based on this view, data can be compressed by overfitting a compact neural network to its functional representation and then encoding the network weights. However, most current solutions for this are inefficient, as quantization to low-bit precision substantially degrades the reconstruction quality. To address this issue, we propose overfitting variational Bayesian neural networks to the data and compressing an approximate posterior weight sample using relative entropy coding instead of quantizing and entropy coding it. This strategy enables direct optimization of the rate-distortion performance by minimizing the $\beta$-ELBO, and target different rate-distortion trade-offs for a given network architecture by adjusting $\beta$. Moreover, we introduce an iterative algorithm for learning prior weight distributions and employ a progressive refinement process for the variational posterior that significantly enhances performance. Experiments show that our method achieves strong performance on image and audio compression while retaining simplicity.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to achieve efficient data compression while maintaining high - quality reconstruction in data compression, especially in the compression of image and audio data. Current methods usually achieve compression by quantizing neural network weights, but this method will significantly reduce the reconstruction quality. To solve this problem, the author proposes a new method based on variational Bayesian implicit neural representations (Bayesian Implicit Neural Representations, INRs), called COMBINER (Compression with Bayesian Implicit Neural Representations). This method samples and encodes directly from the variational posterior distribution of weights by using relative entropy coding (Relative Entropy Coding, REC), instead of quantizing and entropy - coding the weights. This enables direct optimization of the rate - distortion performance and different rate - distortion trade - offs can be achieved by adjusting the parameter \(\beta\).
Specifically, the main contributions of the paper include:
1. **Proposing a new data compression method**: By extending INRs to the variational Bayesian setting, that is, fitting a variational posterior distribution \(q(w)\) to the network weights \(w\), and then using the relative entropy coding algorithm to compress the posterior samples \(w\sim q(w)\).
2. **Proposing an iterative algorithm for learning the prior distribution of weights**: To improve performance, the author develops an iterative algorithm to learn the optimal prior distribution of weights, especially when the variational posterior distribution of INRs is a Gaussian distribution.
3. **Introducing a posterior refinement strategy**: By randomly splitting the weight vector into small pieces and compressing them piece by piece, while performing several gradient descent steps between each encoding step to improve the posterior distribution of the uncompressed weights, thus significantly improving the performance.
Through these improvements, the experimental results of COMBINER on the CIFAR - 10, Kodak image dataset and LibriSpeech audio dataset show that it can achieve strong compression performance while maintaining simplicity and efficiency. In particular, in low - resolution image compression and audio compression, the performance of COMBINER is better than or close to existing state - of - the - art methods, such as COIN++ and MSCN, without the need for a complex meta - learning process.