Abstract:Linear discriminant analysis (LDA) has been a useful tool in pattern recognition and data analysis research and practice. While linearity of class boundaries cannot always be expected, nonlinear projections through pre-trained deep neural networks have served to map complex data onto feature spaces in which linear discrimination has served well. The solution to binary LDA is obtained by eigenvalue analysis of within-class and between-class scatter matrices. It is well known that the multiclass LDA is solved by an extension to the binary LDA, a generalised eigenvalue problem, from which the largest subspace that can be extracted is of dimension one lower than the number of classes in the given problem. In this paper, we show that, apart from the first of the discriminant directions, the generalised eigenanalysis solution to multiclass LDA does neither yield orthogonal discriminant directions nor maximise discrimination of projected data along them. Surprisingly, to the best of our knowledge, this has not been noted in decades of literature on LDA. To overcome this drawback, we present a derivation with a strict theoretical support for sequentially obtaining discriminant directions that are orthogonal to previously computed ones and maximise in each step the Fisher criterion. We show distributions of projections along these axes and demonstrate that discrimination of data projected onto these discriminant directions has optimal separation, which is much higher than those from the generalised eigenvectors of the multiclass LDA. Using a wide range of benchmark tasks, we present a comprehensive empirical demonstration that on a number of pattern recognition and classification problems, the optimal discriminant subspaces obtained by the proposed method, referred to as GO-LDA (Generalised Optimal LDA), can offer superior accuracy.

Optimal projections for Gaussian discriminants

Divergence Maximizing Linear Projection for Supervised Dimension Reduction

A-Optimal Projection for Image Representation.

Contraction rates and projection subspace estimation with Gaussian process priors in high dimension

Which exceptional low-dimensional projections of a Gaussian point cloud can be found in polynomial time?

Proximal methods for sparse optimal scoring and discriminant analysis

Interpolating Discriminant Functions in High-Dimensional Gaussian Latent Mixtures

GO-LDA: Generalised Optimal Linear Discriminant Analysis

Optimizing Allocation Rules in Discrete and Continuous Discriminant Analysis

Classifiability-Based Optimal Discriminatory Projection Pursuit

Optimal low-rank approximations of posteriors for linear Gaussian inverse problems on Hilbert spaces

Quadratic Discriminant Analysis by Projection

Nonstationary Gaussian Process Discriminant Analysis With Variable Selection for High-Dimensional Functional Data

Lorentzian discriminant projection and its applications

A Solution to the Ill-Conditioning of Gradient-Enhanced Covariance Matrices for Gaussian Processes

A novel optimal discriminant principle in high dimensional spaces

Optimal Bayesian design for model discrimination via classification

Optimal Feature Selection in High-Dimensional Discriminant Analysis

Neighborhood MinMax Projections

Discriminant and cluster analysis for Gaussian stationary processes: local linear fitting approach

Optimal Dimensionality Discriminant Analysis and Its Application to Image Recognition