Abstract:Feature selection by maximizing high-order mutual information between the selected feature vector and a target variable is the gold standard in terms of selecting the best subset of relevant features that maximizes the performance of prediction models. However, such an approach typically requires knowledge of the multivariate probability distribution of all features and the target, and involves a challenging combinatorial optimization problem. Recent work has shown that any joint Probability Mass Function (PMF) can be represented as a naive Bayes model, via Canonical Polyadic (tensor rank) Decomposition. In this paper, we introduce a low-rank tensor model of the joint PMF of all variables and indirect targeting as a way of mitigating complexity and maximizing the classification performance for a given number of features. Through low-rank modeling of the joint PMF, it is possible to circumvent the curse of dimensionality by learning principal components of the joint distribution. By indirectly aiming to predict the latent variable of the naive Bayes model instead of the original target variable, it is possible to formulate the feature selection problem as maximization of a monotone submodular function subject to a cardinality constraint which can be tackled using a greedy algorithm that comes with performance guarantees. Numerical experiments with several standard datasets suggest that the proposed approach compares favorably to the state-of-art for this important problem.

Latest CHORUS and NOMAD results

Feature Selection with Conditional Mutual Information Considering Feature Interaction

Feature Selection With Dynamic Mutual Information

Feature Selection Using Mutual Information: An Experimental Study

Class-specific Mutual Information Variation for Feature Selection.

Mutual Information Criteria for Feature Selection

Feature selection based on Mutual Information in supervised learning

Feature Selection by Maximizing Part Mutual Information

Estimating Conditional Mutual Information for Dynamic Feature Selection

Higher Order Mutual Information Approximation for Feature Selection

A Filter Approach to Feature Selection Based on Mutual Information

Beyond Normal: On the Evaluation of Mutual Information Estimators

Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy

Feature selection based on survival Cauchy-Schwartz mutual information

Information-Theoretic Feature Selection via Tensor Decomposition and Submodularity

Input Feature Selection Method Based on Feature Set Equivalence and Mutual Information Gain Maximization

Mutual Information with Parameter Determination Approach for Feature Selection in Multivariate Time Series Prediction.

Input Feature Selection For Sensor Data Mining Based On Mutual Information

Mutual Information Multinomial Estimation

On the Effect of Suboptimal Estimation of Mutual Information in Feature Selection and Classification

A New Approach for Feature Selection from Microarray Data Based on Mutual Information.