On Tsallis Entropy Bias and Generalized Maximum Entropy Models

Yuexian Hou,Tingxu Yan,Peng Zhang,Dawei Song,Wenjie Li

2010-01-01

Abstract:In density estimation task, maximum entropy model (Maxent) can effectivelyuse reliable prior information via certain constraints, i.e., linearconstraints without empirical parameters. However, reliable prior informationis often insufficient, and the selection of uncertain constraints becomesnecessary but poses considerable implementation complexity. Improper setting ofuncertain constraints can result in overfitting or underfitting. To solve thisproblem, a generalization of Maxent, under Tsallis entropy framework, isproposed. The proposed method introduces a convex quadratic constraint for thecorrection of (expected) Tsallis entropy bias (TEB). Specifically, wedemonstrate that the expected Tsallis entropy of sampling distributions issmaller than the Tsallis entropy of the underlying real distribution. Thisexpected entropy reduction is exactly the (expected) TEB, which can beexpressed by a closed-form formula and act as a consistent and unbiasedcorrection. TEB indicates that the entropy of a specific sampling distributionshould be increased accordingly. This entails a quantitative re-interpretationof the Maxent principle. By compensating TEB and meanwhile forcing theresulting distribution to be close to the sampling distribution, ourgeneralized TEBC Maxent can be expected to alleviate the overfitting andunderfitting. We also present a connection between TEB and Lidstone estimator.As a result, TEB-Lidstone estimator is developed by analytically identifyingthe rate of probability correction in Lidstone. Extensive empirical evaluationshows promising performance of both TEBC Maxent and TEB-Lidstone in comparisonwith various state-of-the-art density estimation methods.