Abstract:State-of-the-art high-spectral-efficiency communication systems employ high-order modulation formats coupled with high symbol rates to accommodate the ever-growing demand for data rate-hungry applications. However, such systems are more vulnerable to linear and nonlinear transmission impairments, and it is important to mitigate the performance loss via digital signal processing. In this paper, we propose a novel machine learning approach for blind channel equalization and estimation using the vector quantized (VQ) \ac{VAE} framework. The proposed approach generalizes the applicability of the conventional \ac{VAE}-based equalizer to nonlinear systems employing high-order modulation formats by introducing a codebook component and an associated novel loss function. We evaluate the performance of the proposed method over a linear additive white Gaussian noise channel with intersymbol interference and two nonlinear scenarios. Simulation results show that the proposed method can achieve similar performance as a data aided equalizer using the \acf{MMSE} criterion, and outperforms the blind\ac{CMA} and the \ac{VAE}-based channel equalizer. Furthermore, we show that for the linear channel, the proposed scheme exhibits better convergence properties than the \ac{MMSE}-based, the \ac{CMA}-based, and the \ac{VAE}-based equalizers in terms of both convergence speed and robustness to variations in training batch size and learning rate.

Wavelet Neural Networks for Adaptive Equalization

Wavelet Neural Networks for Adaptive Equalization by Using the Orthogonal Least Square Algorithm

Study on Channel Equalization Based on RBF Networks

A Multilayer Complex Neural Network Training Algorithm and Its Application in Adaptive Equalization

NeuralEQ: Neural-Network-Based Equalizer for High-Speed Wireline Communication

Nonlinear Equalization for TDMR Channels Using Neural Networks

Weight-adaptive joint mixed-precision quantization and pruning for neural network-based equalization in short-reach direct detection links

Joint Neural Network Equalizer and Decoder

Improved Weighted Multimodulus Blind Equalization Algorithm

Blind Channel Equalization Using Vector-Quantized Variational Autoencoders

Discrete wavelet transform assisted convolutional neural network equalizer for PAM VLC system

Artificial neural network-based nonlinear channel equalization: A soft-output perspective

Adaptive and Efficient Nonlinear Channel Equalization for Underwater Acoustic Communication

Adaptive Turbo Equalization for Nonlinearity Compensation in WDM Systems

Wide and Deep Learning-Aided Nonlinear Equalizer for Coherent Optical Communication Systems

Low Complexity Joint Neural Network Equalizer in a 248 Gbit/s VSB PS-PAM8 IM/DD Transmission System

Experimental Study of Deep Neural Network Equalizers Performance in Optical Links

Blind and Channel-agnostic Equalization Using Adversarial Networks

Neural Network-based Equalizer by Utilizing Coding Gain in Advance

Blind Fuzzy Adaptation Step Control for a Concurrent Neural Network Equalizer