Abstract:Maximum entropy approach to classification is very well studied in applied statistics and machine learning and almost all the methods that exists in literature are discriminative in nature. In this paper, we introduce a maximum entropy classification method with feature selection for large dimensional data such as text datasets that is generative in nature. To tackle the curse of dimensionality of large data sets, we employ conditional independence assumption (Naive Bayes) and we perform feature selection simultaneously, by enforcing a ‘maximum discrimination’ between estimated class conditional densities. For two class problems, in the proposed method, we use Jeffreys ($(J)$ divergence to discriminate the class conditional densities. To extend our method to the multi-class case, we propose a completely new approach by considering a multi-distribution divergence: we replace Jeffreys divergence by Jensen-Shannon $(JS)$ divergence to discriminate conditional densities of multiple classes. In order to reduce computational complexity, we employ a modified Jensen-Shannon divergence $(JS_{GM}$, based on AM-GM inequality. We show that the resulting divergence is a natural generalization of Jeffreys divergence to a multiple distributions case. As far as the theoretical justifications are concerned we show that when one intends to select the best features in a generative maximum entropy approach, maximum discrimination using $J$-divergence emerges naturally in binary classification. Performance and comparative study of the proposed algorithms have been demonstrated on large dimensional text and gene expression datasets that show our methods scale up very well with large dimensional datasets.

Generalized Maximum Entropy for Supervised Classification

Generalized entropy based semi-supervised learning

The Principle of Uncertain Maximum Entropy

Minimax risk classifiers with 0-1 loss

Max-margin Based Bayesian Classifier.

Generative Maximum Entropy Learning for Multiclass Classification

Multi-labelled classification using maximum entropy method.

Supervised classification via minimax probabilistic transformations

An Extended Result on the Optimal Estimation under the Minimum Error Entropy Criterion

Understanding Deep Learning Generalization by Maximum Entropy

Robust Discriminant Analysis Based on Nonparametric Maximum Entropy

Restricted Minimum Error Entropy Criterion for Robust Classification

Semi-supervised Multi-View Maximum Entropy Discrimination with Expectation Laplacian Regularization

High-dimensional logistic entropy clustering

Some Further Results on the Minimum Error Entropy Estimation.

On the Smoothed Minimum Error Entropy Criterion.

Maximum entropy model and conditional random field

Maximum Multiscale Entropy and Neural Network Regularization

If MaxEnt RL is the Answer, What is the Question?

Generalised Entropy MDPs and Minimax Regret

General Multi-View Learning with Maximum Entropy Discrimination