Abstract:In density estimation task, maximum entropy model (Maxent) can effectivelyuse reliable prior information via certain constraints, i.e., linearconstraints without empirical parameters. However, reliable prior informationis often insufficient, and the selection of uncertain constraints becomesnecessary but poses considerable implementation complexity. Improper setting ofuncertain constraints can result in overfitting or underfitting. To solve thisproblem, a generalization of Maxent, under Tsallis entropy framework, isproposed. The proposed method introduces a convex quadratic constraint for thecorrection of (expected) Tsallis entropy bias (TEB). Specifically, wedemonstrate that the expected Tsallis entropy of sampling distributions issmaller than the Tsallis entropy of the underlying real distribution. Thisexpected entropy reduction is exactly the (expected) TEB, which can beexpressed by a closed-form formula and act as a consistent and unbiasedcorrection. TEB indicates that the entropy of a specific sampling distributionshould be increased accordingly. This entails a quantitative re-interpretationof the Maxent principle. By compensating TEB and meanwhile forcing theresulting distribution to be close to the sampling distribution, ourgeneralized TEBC Maxent can be expected to alleviate the overfitting andunderfitting. We also present a connection between TEB and Lidstone estimator.As a result, TEB-Lidstone estimator is developed by analytically identifyingthe rate of probability correction in Lidstone. Extensive empirical evaluationshows promising performance of both TEBC Maxent and TEB-Lidstone in comparisonwith various state-of-the-art density estimation methods.

Improving Decision Trees by Tsallis Entropy Information Metric Method.

A less-greedy two-term Tsallis Entropy Information Metric approach for decision tree classification

Unifying Decision Trees Split Criteria Using Tsallis Entropy

Unifying Attribute Splitting Criteria of Decision Trees by Tsallis Entropy.

Rough set based decision tree

An Improved Belief Entropy and Its Application in Decision-Making

A New Approach of the Split Criteria of Decision Trees Based on the T-entropy

An Improved Algorithm of Decision Tree Based on Attribute Entropy

Automatic decision support by information energy decision tree algorithm.

An Improvement On The Algorithm Of Decision Tree

Improved Information Gain Estimates for Decision Tree Induction

Learning Decision Trees Using Confusion Entropy

Evidential Decision Tree Based on Belief Entropy

Rank Entropy-Based Decision Trees for Monotonic Classification

Analysis of the Credible Degree of the Decision Based on Information Entropy

A Spatial Entropy-Based Decision Tree for Classification of Geographical Information

A modified ID3 decision tree algorithm based on cumulative residual entropy

A Generalized Fuzzy ID3 Algorithm Using Generalized Information Entropy

Performance Improvement Validation of Decision Tree Algorithms with Non-normalized Information Distance in Experiments

Random forests with parametric entropy-based information gains for classification and regression problems

On Tsallis Entropy Bias and Generalized Maximum Entropy Models