Abstract:Covariance and Hessian matrices have been analyzed separately in the literature for classification problems. However, integrating these matrices has the potential to enhance their combined power in improving classification performance. We present a novel approach that combines the eigenanalysis of a covariance matrix evaluated on a training set with a Hessian matrix evaluated on a deep learning model to achieve optimal class separability in binary classification tasks. Our approach is substantiated by formal proofs that establish its capability to maximize between-class mean distance (the concept of \textit{separation}) and minimize within-class variances (the concept of \textit{compactness}), which together define the two linear discriminant analysis (LDA) criteria, particularly under ideal data conditions such as isotropy around class means and dominant leading eigenvalues. By projecting data into the combined space of the most relevant eigendirections from both matrices, we achieve optimal class separability as per these LDA criteria. Empirical validation across neural and health datasets consistently supports our theoretical framework and demonstrates that our method outperforms established methods. Our method stands out by addressing both separation and compactness criteria, unlike PCA and the Hessian method, which predominantly emphasize one criterion each. This comprehensive approach captures intricate patterns and relationships, enhancing classification performance. Furthermore, through the utilization of both LDA criteria, our method outperforms LDA itself by leveraging higher-dimensional feature spaces, in accordance with Cover's theorem, which favors linear separability in higher dimensions. Additionally, our approach sheds light on complex DNN decision-making, rendering them comprehensible within a 2D space.

Maximizing adjusted covariance: new supervised dimension reduction for classification

Computational and Theoretical Analysis of Supervised Dimensionality Reduction

Laplacian MinMax Discriminant Analysis and Its Applications

Discriminative Dimension Reduction Via Maximin Separation Probability Analysis

Efficient Maximal Coding Rate Reduction by Variational Forms

Maximally Correlated Principal Component Analysis

A Variance Minimization Criterion to Feature Selection Using Laplacian Regularization

A New Covariance Estimator for Sufficient Dimension Reduction in High-Dimensional and Undersized Sample Problems

Dimension Reduction with Semi-supervised Pairwise Covariance-Preserving Projection

Laplacian MinMax Discriminant Projection and Its Applications.

Supervised Linear Dimension-Reduction Methods: Review, Extensions, and Comparisons

Bayesian Maximum Margin Principal Component Analysis

Dimension Reduction by Minimum Error Minimax Probability Machine

Divergence Maximizing Linear Projection for Supervised Dimension Reduction

Synergistic eigenanalysis of covariance and Hessian matrices for enhanced binary classification

Reconstructable generalized maximum scatter difference discriminant analysis

Functional sufficient dimension reduction through distance covariance

Nonparametric Dimension Reduction Via Maximizing Pairwise Separation Probability.

Dimension Reduction by Maximizing Pairwise Discriminations.

Improved Covariance Modeling For Maxim Unilikelihood Multiple Subspace Transformations

Distance metric learning by minimal distance maximization