Abstract:Recent studies suggest that the minimum error entropy (MEE) criterion can outperform the traditional mean square error criterion in supervised machine learning, especially in nonlinear and non-Gaussian situations. In practice, however, one has to estimate the error entropy from the samples since in general the analytical evaluation of error entropy is not possible. By the Parzen windowing approach, the estimated error entropy converges asymptotically to the entropy of the error plus an independent random variable whose probability density function (PDF) corresponds to the kernel function in the Parzen method. This quantity of entropy is called the smoothed error entropy, and the corresponding optimality criterion is named the smoothed MEE (SMEE) criterion. In this paper, we study theoretically the SMEE criterion in supervised machine learning where the learning machine is assumed to be nonparametric and universal. Some basic properties are presented. In particular, we show that when the smoothing factor is very small, the smoothed error entropy equals approximately the true error entropy plus a scaled version of the Fisher information of error. We also investigate how the smoothing factor affects the optimal solution. In some special situations, the optimal solution under the SMEE criterion does not change with increasing smoothing factor. In general cases, when the smoothing factor tends to infinity, minimizing the smoothed error entropy will be approximately equivalent to minimizing error variance, regardless of the conditional PDF and the kernel.

A Dynamic Parzen Window Approach Based on Error-entropy Minimization Algorithm for Supervised Training of Nonlinear Adaptive System

An Adaptive System Identification Algorithm With A General Performace Index Based On Entropy Optimization

An Adaptive System Identification Algorithm with a General Performance Index Based on Entropy Optimization

Minimum Total Error Entropy Method for Parameter Estimation

Adaptive Learning Algorithms with Fast Convergence Rate for Minimum Bit Error Rate Nonlinear Equalizers

Mean-square convergence analysis of ADALINE training with minimum error entropy criterion.

A Dual Perspective of Sparse and Robust Online Learning Algorithm

Robust Kernel Recursive Adaptive Filtering Algorithms Based on M-estimate

Robust Adaptive Filtering based on M-estimation-based Minimum Error Entropy Criterion

Parameter Estimation for Nonlinear Stochastic Model Using Generalized Entropy Optimization Principle

On the Smoothed Minimum Error Entropy Criterion.

Robust Constrained Adaptive Filtering under Minimum Error Entropy Criterion

Nonlinear Recursive Minimum Model Error Estimation

Adaptive Soft Sensor Modeling Based on Weighted Supervised Latent Factor Analysis with Selectively Integrated Moving Windows

Kernel Minimum Error Entropy Algorithm

Proportionate Minimum Error Entropy Algorithm for Sparse System Identification

Mixture Quantized Error Entropy for Recursive Least Squares Adaptive Filtering

Minimum Error Entropy Algorithms With Sparsity Penalty Constraints

A kernel recursive minimum error entropy adaptive filter

Multipath Estimation Using Kernel Minimum Error Entropy Filter