Abstract:Recently, the minimum error entropy criterion, an information theoretic alternative to the traditional mean square error criterion, has been successfully used in the contexts of machine learning and signal processing. For system identification, however, the MEE criterion will be no longer suitable if the training data are discrete-valued, since minimizing error’s discrete entropy cannot constrain error’s dispersion. In this paper, to make the MEE criterion suitable for the discrete-valued data cases, we give a new entropy definition for the discrete random variables, i.e. the Δ-entropy, based on Riemann sums for finite size partitions. A probability weighted formula is established to calculate the average partition.This new entropy retains some important properties of the differential entropy and reduces to discrete entropy under certain conditions. Unlike discrete entropy, the Δ-entropy is sensitive to the dynamic range of the data, and can be used as a superior optimality criterion in system identification problems. Also, we present a plug-in estimate of Δ-entropy, analyze its asymptotic behavior and explore the links to the kernel based and m-spacing based estimates for differential entropy. Finally, the Δ-entropy criterion is applied in system identification with coarsely quantized input–output data to search for the optimum parameter set. Monte Carlo simulations demonstrate the performance improvement that may be achieved with the Δ-entropy criterion.

Epsilon-entropy rate and Bode's sensitivity integral in continuous time systems

Epsilon-Entropy and H∞ Entropy in Continuous Time Systems

Variety of linear continuous time systems

Entropy Rate and H∞ Entropy in LTI Control Systems

On MMSE, Mutual Information and Popov Function in Continuous-Time Multivariable Linear Systems

Information Theoretic Interpretation for H∞ Entropy in SISO Linear Control Systems

A Robust Output Error Identifier for Continuous-Time Systems

Information theoretic interpretations for H˞ entropy

On Mutual Information, Mmse And H-Infinity Entropy In Stochastic Linear Feedback Systems

An information theoretic approach to performance limits in linear time invariant control systems

A New H-2 Bound For H-Infinity Entropy

An Information Theoretic Interpretation for H-Infinity Entropy

Performance Limits of Disturbance Rejection in Linear Multivariable Control Systems:Information Theoretic Approaches

K-L Information Rate and H/sub ∞/ Entropy in Linear Control Systems

Bode Integrals and Laws of Variety in Linear Control Systems

Sensitivity Analysis of Continuous-Time Systems based on Power Spectral Density

H∞ Entropy and the Law of Requisite Variety

Mutual Information Rate and Analytic Constraint in Linear Time Invariant Control Systems

Δ-Entropy: Definition, Properties and Applications in System Identification with Quantized Data

State Estimation of Continuous-time Dynamical Systems with Uncertain Inputs with Bounded Variation: Entropy, Bit Rates, and Relation with Switched Systems

An Information-Theoretic Analysis of Discrete-Time Control and Filtering Limitations by the I-MMSE Relationships