Δ-Entropy: Definition, Properties and Applications in System Identification with Quantized Data
Badong Chen,Yu Zhu,Jinchun Hu,Jose C. Principe
DOI: https://doi.org/10.1016/j.ins.2010.11.037
IF: 8.1
2010-01-01
Information Sciences
Abstract:Recently, the minimum error entropy criterion, an information theoretic alternative to the traditional mean square error criterion, has been successfully used in the contexts of machine learning and signal processing. For system identification, however, the MEE criterion will be no longer suitable if the training data are discrete-valued, since minimizing error’s discrete entropy cannot constrain error’s dispersion. In this paper, to make the MEE criterion suitable for the discrete-valued data cases, we give a new entropy definition for the discrete random variables, i.e. the Δ-entropy, based on Riemann sums for finite size partitions. A probability weighted formula is established to calculate the average partition.This new entropy retains some important properties of the differential entropy and reduces to discrete entropy under certain conditions. Unlike discrete entropy, the Δ-entropy is sensitive to the dynamic range of the data, and can be used as a superior optimality criterion in system identification problems. Also, we present a plug-in estimate of Δ-entropy, analyze its asymptotic behavior and explore the links to the kernel based and m-spacing based estimates for differential entropy. Finally, the Δ-entropy criterion is applied in system identification with coarsely quantized input–output data to search for the optimum parameter set. Monte Carlo simulations demonstrate the performance improvement that may be achieved with the Δ-entropy criterion.