Abstract:We present a subspace method to solve large-scale trace ratio problems. This method is matrix-free, only needing the action of the two matrices in the trace ratio. At each iteration, a smaller trace ratio problem is addressed in the search subspace. Additionally, our algorithm is endowed with a restarting strategy, that ensures the monotonicity of the trace ratio value throughout the iterations. We also investigate the behavior of the approximate solution from a theoretical viewpoint, extending existing results on Ritz values and vectors, as the angle between the search subspace and the exact solution to the trace ratio approaches zero. In the context of multigroup classification, numerical experiments show that the new subspace method tends to be more efficient than iterative approaches that need a (partial) eigenvalue decomposition in every step.

What problem does this paper attempt to address?

This paper attempts to solve the computational efficiency and precision problems of large - scale trace ratio problems. Specifically, the author proposes a subspace method to solve the large - scale trace ratio optimization problem. This method only requires matrix - vector multiplication in each iteration, without directly handling the complete eigenvalue problem. This makes the method more efficient when dealing with large - scale data. ### Problem Background The Trace Ratio Problem (TR) can be expressed as: \[ \max_{V^TV = I_k}\rho(V):=\frac{\text{tr}(V^TAV)}{\text{tr}(V^TBV)} \] where \(A\) is a symmetric matrix, \(B\) is a symmetric positive definite matrix, and \(V\) is a \(p\times k\) matrix whose column vectors are orthogonal. The trace ratio problem has important applications in linear dimensionality reduction and feature selection in multi - group classification, such as Fisher Discriminant Analysis (FDA). ### Main Contributions of the Paper 1. **Subspace Method**: The author proposes a subspace method. This method gradually approximates the solution of the trace ratio problem by projecting matrices \(A\) and \(B\) onto lower - dimensional subspaces. In each iteration, the algorithm solves a small - scale trace ratio problem in the search subspace and uses the residual matrix to expand the search subspace. 2. **Restart Strategy**: To reduce the computational cost and memory requirements, the algorithm introduces a restart strategy. When the dimension of the search subspace reaches the preset maximum value, the algorithm will reduce the dimension of the subspace by retaining the most important information, thus ensuring that the trace ratio value monotonically increases throughout the iteration process. 3. **Theoretical Analysis**: The author theoretically analyzes the behavior of the approximate solution, especially when the angle between the search subspace and the exact solution approaches zero. They extend the existing results on Ritz values and vectors and prove the effectiveness and convergence of the algorithm. 4. **Numerical Experiments**: Through experiments on synthetic and real - world datasets, the author shows the superior performance of the newly proposed subspace method in multi - group classification tasks, especially being more efficient in iterative methods that require partial eigenvalue decomposition. ### Key Formulas - The objective function of the trace ratio problem: \[ \rho(V)=\frac{\text{tr}(V^TAV)}{\text{tr}(V^TBV)} \] - Residual matrix: \[ R = AV-\rho_jBV - V\Lambda \] where \(\Lambda = V^T(A - \rho_jB)V\) is a diagonal matrix containing the largest \(k\) eigenvalues of \(H_j-\rho_jK_j\). - Trace ratio problem after subspace projection: \[ \rho_j=\max_{Z^TZ = I_k}\frac{\text{tr}(Z^TH_jZ)}{\text{tr}(Z^TK_jZ)} \] where \(H_j = U_j^TAU_j\) and \(K_j = U_j^TBU_j\). ### Summary This paper proposes a novel subspace method to solve the large - scale trace ratio problem, which is especially suitable for linear dimensionality reduction tasks in multi - group classification. Through an effective restart strategy and theoretical analysis, this method not only improves computational efficiency but also ensures the stability and convergence of the algorithm.

A subspace method for large-scale trace ratio problems

A Convergence Analysis on the Iterative Trace Ratio Algorithm and Its Refinements

Trace Ratio Problem Revisited

Computation of the von Neumann entropy of large matrices via trace estimators and rational Krylov methods

Trace Ratio Optimization with an Application to Multi-view Learning

Trace-Penalty Minimization for Large-Scale Eigenspace Computation

Perturbation Analysis for the Trace Quotient Problem

A Refined Subspace Iteration Algorithm For Large Sparse Eigenproblems

A Note on the Trace Quotient Problem.

A Convex Programming Approach To The Trace Quotient Problem

A Riemannian Subspace Limited-Memory SR1 Trust Region Method

Closing the duality gap of the generalized trace ratio problem

Double-variable trace maximization for extreme generalized singular quartets of a matrix pair: A geometric method

A Novel Formulation of Trace Ratio Linear Discriminant Analysis

Analysis of stochastic probing methods for estimating the trace of functions of sparse symmetric matrices

Subspace Methods with Local Refinements for Eigenvalue Computation Using Low-Rank Tensor-Train Format.

A Riemannian Subspace BFGS Trust Region Method

Rayleigh-Ritz majorization error bounds with applications to FEM

Tractable and Scalable Schatten Quasi-Norm Approximations for Rank Minimization

Nonparametric Trace Regression in High Dimensions via Sign Series Representation

Safe subspace screening for the adaptive nuclear norm regularized trace regression