Abstract:Linear discriminant analysis (LDA) is a classical method for dimensionality reduction, where discriminant vectors are sought to project data to a lower dimensional space for optimal separability of classes. Several recent papers have outlined strategies, based on exploiting sparsity of the discriminant vectors, for performing LDA in the high-dimensional setting where the number of features exceeds the number of observations in the data. However, many of these proposed methods lack scalable methods for solution of the underlying optimization problems. We consider an optimization scheme for solving the sparse optimal scoring formulation of LDA based on block coordinate descent. Each iteration of this algorithm requires an update of a scoring vector, which admits an analytic formula, and an update of the corresponding discriminant vector, which requires solution of a convex subproblem; we will propose several variants of this algorithm where the proximal gradient method or the alternating direction method of multipliers is used to solve this subproblem. We show that the per-iteration cost of these methods scales linearly in the dimension of the data provided restricted regularization terms are employed, and cubically in the dimension of the data in the worst case. Furthermore, we establish that when this block coordinate descent framework generates convergent subsequences of iterates, then these subsequences converge to the stationary points of the sparse optimal scoring problem. We demonstrate the effectiveness of our new methods with empirical results for classification of Gaussian data and data sets drawn from benchmarking repositories, including time-series and multispectral X-ray data, and provide Matlab and R implementations of our optimization schemes.

Worst-Case Linear Discriminant Analysis As Scalable Semidefinite Feasibility Problems

A Bi-level Nonlinear Eigenvector Algorithm for Wasserstein Discriminant Analysis

Towards Scalable Semidefinite Programming: Optimal Metric ADMM with A Worst-case Performance Guarantee

Understanding and Evaluating Sparse Linear Discriminant Analysis

Proximal methods for sparse optimal scoring and discriminant analysis

Robust Linear Discriminant Analysis Using Ratio Minimization of L1,2-Norms

SRDA: an Efficient Algorithm for Large-Scale Discriminant Analysis

Sparse Exponential Discriminant Analysis

Sr-LDA:Sparse and Reduced-Rank Linear Discriminant Analysis for High Dimensional Matrix

Making Fisher Discriminant Analysis Scalable.

Optimal Feature Selection for Sparse Linear Discriminant Analysis and Its Applications in Gene Expression Data

SRDA: an Efficient Algorithm for Large-Scale

Least Square Incremental Linear Discriminant Analysis

L2,1-Norm Regularized Robust and Sparse Linear Discriminant Analysis via an Alternating Direction Method of Multipliers

Quadratic Discriminant Analysis by Projection

An Efficient Greedy Search Algorithm for High-Dimensional Linear Discriminant Analysis

Least squares online linear discriminant analysis

Two‐Stage Estimation for Ultrahigh Dimensional Sparse Quadratic Discriminant Analysis

Separable Two-Dimensional Linear Discriminant Analysis

Linear and Deep Order-Preserving Wasserstein Discriminant Analysis

Suboptimality bounds for trace-bounded SDPs enable a faster and scalable low-rank SDP solver SDPLR+