Abstract:State-of-the-art high-spectral-efficiency communication systems employ high-order modulation formats coupled with high symbol rates to accommodate the ever-growing demand for data rate-hungry applications. However, such systems are more vulnerable to linear and nonlinear transmission impairments, and it is important to mitigate the performance loss via digital signal processing. In this paper, we propose a novel machine learning approach for blind channel equalization and estimation using the vector quantized (VQ) \ac{VAE} framework. The proposed approach generalizes the applicability of the conventional \ac{VAE}-based equalizer to nonlinear systems employing high-order modulation formats by introducing a codebook component and an associated novel loss function. We evaluate the performance of the proposed method over a linear additive white Gaussian noise channel with intersymbol interference and two nonlinear scenarios. Simulation results show that the proposed method can achieve similar performance as a data aided equalizer using the \acf{MMSE} criterion, and outperforms the blind\ac{CMA} and the \ac{VAE}-based channel equalizer. Furthermore, we show that for the linear channel, the proposed scheme exhibits better convergence properties than the \ac{MMSE}-based, the \ac{CMA}-based, and the \ac{VAE}-based equalizers in terms of both convergence speed and robustness to variations in training batch size and learning rate.

Blind Equalization of Multilevel Signals Via Support Vector Regression.

Excess MSE Analysis of the Concurrent Constant Modulus Algorithm and Soft Decision-Directed Scheme for Blind Equalisation

Blind Equalization Algorithm for Underwater Acoustic Channel Based on Support Vector Regression

Blind Equalization Algorithm of QAM Signal Based on SNR

Blind Equalization of Constant Modulus Based on Support Vector Regression

A New Dual Mode Blind Equalization Algorithm

A Integrated Algorithm of DDLMS and Godard for Blind Equilization

A Variable Step Size Blind Equalization Algorithm for QAM Signals

Implementation of blind equalization algorithm for HF QAM based on DSP

Improved Weighted Multimodulus Blind Equalization Algorithm

A Stop and Go Blind Equalization Algorithm for QAM Signals

Dual Mode Blind Equalization Algorithm Based on ATSC DTV Receiver

New Blind Equalisation for Non-Constant Modulus Signals Using a Segment Cost Function

A Set-Membership-Based Blind Equalization Algorithm for High-order QAM Signals

Variable Step-size Blind Equalization Algrothm Based on Multi-Mode Error Switch

Joint CMA + DDLMS Blind Equalization Algorithm

Blind Equalization of MIMO Systems Based on Orthogonal Constant Modulus Algorithm

Blind Channel Equalization Using Vector-Quantized Variational Autoencoders

Semidefinite Relaxation Based Blind Equalization using Constant Modulus Criterion

Blind Channel Equalization