Neural Distributed Compressor Discovers Binning

Ezgi Ozyilkan,Johannes Ballé,Elza Erkip

DOI: https://doi.org/10.1109/JSAIT.2024.3393429

2023-10-26

Abstract:We consider lossy compression of an information source when the decoder has lossless access to a correlated one. This setup, also known as the Wyner-Ziv problem, is a special case of distributed source coding. To this day, practical approaches for the Wyner-Ziv problem have neither been fully developed nor heavily investigated. We propose a data-driven method based on machine learning that leverages the universal function approximation capability of artificial neural networks. We find that our neural network-based compression scheme, based on variational vector quantization, recovers some principles of the optimum theoretical solution of the Wyner-Ziv setup, such as binning in the source space as well as optimal combination of the quantization index and side information, for exemplary sources. These behaviors emerge although no structure exploiting knowledge of the source distributions was imposed. Binning is a widely used tool in information theoretic proofs and methods, and to our knowledge, this is the first time it has been explicitly observed to emerge from data-driven learning.

Information Theory,Signal Processing

What problem does this paper attempt to address?

The paper attempts to address the problem of how to effectively perform lossy compression in Distributed Source Coding (DSC) when the decoder can losslessly access a related auxiliary information. Specifically, the paper focuses on the Wyner-Ziv problem, which is a special DSC scenario where the encoder compresses the source data, and the decoder, while receiving the compressed data, can also access an auxiliary information related to the source data. Although it has been theoretically proven that optimal compression efficiency can be achieved in this case, the methods in practical applications are still not mature and effective.

The main contribution of the paper is the proposal of a machine learning-based method that utilizes the universal function approximation capability of Artificial Neural Networks (ANNs) to design a new compression scheme. This scheme can automatically learn the optimal compression strategy without prior knowledge of the joint distribution of the source data and the auxiliary information, including the binning mechanism in the source space and the optimal combination of quantization indices and auxiliary information. These behaviors are achieved through Variational Vector Quantization (VVQ) techniques, and the experiments demonstrate behaviors consistent with the theoretically optimal solution. This is the first time that the emergence of the binning mechanism has been clearly observed from data-driven learning.