Abstract:In this paper, we study a remote source coding scenario in which binary phase shift keying (BPSK) modulation sources are corrupted by additive white Gaussian noise (AWGN). An intermediate node, such as a relay, receives these observations and performs additional compression to balance complexity and relevance. This problem can be further formulated as an information bottleneck (IB) problem with Bernoulli sources and Gaussian mixture observations. However, no closed-form solution exists for this IB problem. To address this challenge, we propose a unified achievable scheme that employs three different compression/quantization strategies for intermediate node processing by using two-level quantization, multi-level deterministic quantization, and soft quantization with the hyperbolic tangent ($\tanh$) function, respectively. In addition, we extend our analysis to the vector mixture Gaussian observation problem and explore its application in machine learning for binary classification with information leakage. Numerical evaluations show that the proposed scheme has a near-optimal performance over various signal-to-noise ratios (SNRs), compared to the Blahut-Arimoto (BA) algorithm, and has better performance than some existing numerical methods such as the information dropout approach. Furthermore, experiments conducted on the realistic MNIST dataset also validate the superior classification accuracy of our method compared to the information dropout approach.

What problem does this paper attempt to address?

This paper mainly discusses the solutions to the Information Bottleneck (IB) problem under Gaussian mixture observation. The Information Bottleneck is an optimization framework used to extract the most important information about the target variable from the observed data while compressing the data to reduce redundancy. In this paper, the authors consider a remote source coding scenario where binary phase-shift keying (BPSK) signals are interfered by additive white Gaussian noise (AWGN) channels, and an intermediate node receives these observations and further compresses them. The main contributions of the paper are as follows: 1. A unified and implementable solution is proposed, which uses two levels of quantization, multi-level deterministic quantization, and soft quantization strategy with a hyperbolic tangent function, to tackle the Gaussian mixture observation information bottleneck problem. These methods demonstrate different performances in different complexity and correlation tradeoff curves. 2. The analysis is extended to the vector Gaussian mixture observation problem, expanding the application scope of the framework. 3. The application of the information bottleneck in binary classification problems with information leakage is explored, and the proposed method outperforms the information dropout method in terms of classification accuracy on the MNIST dataset. The paper first introduces the basic concept of the information bottleneck and its applications in communication and machine learning. Then it discusses the information bottleneck problem of Gaussian mixture models, which currently can only be solved through numerical algorithms. The authors propose three new analytical and implementable methods to solve this problem and demonstrate their superiority through comparisons with the Blahut-Arimoto algorithm and other numerical methods. Finally, numerical results on real datasets are presented to validate the effectiveness of the proposed methods.

An Achievable and Analytic Solution to Information Bottleneck for Gaussian Mixtures

The Double-Sided Information Bottleneck Function

Distributed Information Bottleneck for a Primitive Gaussian Diamond Channel with Rayleigh Fading

On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views

Deep-Learning Based Gaussian Mixture Likelihoods Joint Source-Channel Coding for Image Transmission.

Gray--Wyner and Mutual Information Regions for Doubly Symmetric Binary Sources and Gaussian Sources

Broadcast Channel with Transmitter Noncausal Interference and Receiver Side Information.

Information Bottleneck Revisited: Posterior Probability Perspective with Optimal Transport

Semantic-Aware Multi-Terminal Coding for Gaussian Mixture Sources

Generalized Information Bottleneck for Gaussian Variables

Neural Network Activation Quantization with Bitwise Information Bottlenecks

Efficient and Provably Convergent Computation of Information Bottleneck: A Semi-Relaxed Approach

Hybrid Digital-Analog Codes for Source-Channel Broadcast of Gaussian Sources Over Gaussian Channels

Energy Efficient Estimation of Gaussian Sources Over Inhomogeneous Gaussian MAC Channels

Information Bottleneck for a Rayleigh Fading MIMO Channel with an Oblivious Relay

Opportunistic Information-Bottleneck for Goal-oriented Feature Extraction and Communication

Information Bottleneck Methods for Distributed Learning

GaussianSource Coding Based on P-LDPC Code

Communication under Mixed Gaussian-Impulsive Channel: an End-to-End Framework

PAC-Bayes Information Bottleneck

Broadcast Approach for the Information Bottleneck Channel