Abstract:Linear discriminant analysis (LDA) is a classical method for dimensionality reduction, where discriminant vectors are sought to project data to a lower dimensional space for optimal separability of classes. Several recent papers have outlined strategies, based on exploiting sparsity of the discriminant vectors, for performing LDA in the high-dimensional setting where the number of features exceeds the number of observations in the data. However, many of these proposed methods lack scalable methods for solution of the underlying optimization problems. We consider an optimization scheme for solving the sparse optimal scoring formulation of LDA based on block coordinate descent. Each iteration of this algorithm requires an update of a scoring vector, which admits an analytic formula, and an update of the corresponding discriminant vector, which requires solution of a convex subproblem; we will propose several variants of this algorithm where the proximal gradient method or the alternating direction method of multipliers is used to solve this subproblem. We show that the per-iteration cost of these methods scales linearly in the dimension of the data provided restricted regularization terms are employed, and cubically in the dimension of the data in the worst case. Furthermore, we establish that when this block coordinate descent framework generates convergent subsequences of iterates, then these subsequences converge to the stationary points of the sparse optimal scoring problem. We demonstrate the effectiveness of our new methods with empirical results for classification of Gaussian data and data sets drawn from benchmarking repositories, including time-series and multispectral X-ray data, and provide Matlab and R implementations of our optimization schemes.

The Minimum $L_2$-Distance Projection onto the Canonical Simplex: A Simple Algorithm

Fast projection onto the intersection of simplex and singly linear constraint and its generalized Jacobian

Projections in L2 and L1 normed spaces and SVDs

Convex optimization over a probability simplex

Single-Projection Procedure for Infinite Dimensional Convex Optimization Problems

Semismooth Newton Algorithm for Efficient Projections Onto ℓ1, ∞-Norm Ball

Sparse Multiple Kernel Learning: Minimax Rates with Random Projection

Distance Majorization and Its Applications

Convergence Rate Analysis of a Dykstra-Type Projection Algorithm

Semismooth Newton Algorithm for Efficient Projections onto $\ell_1, ∞$-norm Ball

Laplacian MinMax Discriminant Projection and Its Applications.

The Minimum Norm of a Projector under Linear Interpolation on a Euclidean Ball

A Purely Geometric Approach to the Problem of Computing the Projection of a Point on a Simplex

Proximal methods for sparse optimal scoring and discriminant analysis

Mixed-Projection Conic Optimization: A New Paradigm for Modeling Rank Constraints

Inertial projection methods for finding a minimum-norm solution of pseudomonotone variational inequality and fixed-point problems

Sparsity-Exploiting Distributed Projections onto a Simplex

Neighborhood MinMax Projections

nSimplex Zen: A Novel Dimensionality Reduction for Euclidean and Hilbert Spaces