Abstract:The convergence of Oja's principal component analysis (PCA) learning algorithms is a difficult topic for direct study and analysis. Traditionally, the convergence of these algorithms is indirectly analyzed via certain deterministic continuous time (DCT) systems. Such a method will require the learning rate to converge to zero, which is not a reasonable requirement to impose in many practical applications. Recently, deterministic discrete time (DDT) systems have been proposed instead to indirectly interpret the dynamics of the learning algorithms. Unlike DCT systems, DDT systems allow learning rates to be constant (which can be a nonzero). This paper will provide some important results relating to the convergence of a DDT system of Oja's PCA learning algorithm. It has the following contributions: 1) A number of invariant sets are obtained, based on which we can show that any trajectory starting from a point in the invariant set will remain in the set forever. Thus, the nondivergence of the trajectories is guaranteed. 2) The convergence of the DDT system is analyzed rigorously. It is proven, in the paper, that almost all trajectories of the system starting from points in an invariant set will converge exponentially to the unit eigenvector associated with the largest eigenvalue of the correlation matrix. In addition, exponential convergence rate are obtained, providing useful guidelines for the selection of fast convergence learning rate. 3) Since the trajectories may diverge, the careful choice of initial vectors is an important issue. This paper suggests to use the domain of unit hyper sphere as initial vectors to guarantee convergence. 4) Simulation results will be furnished to illustrate the theoretical results achieved.

Online Learning for Sparse PCA in High Dimensions: Exact Dynamics and Phase Transitions

Langevin dynamics for high-dimensional optimization: the case of multi-spiked tensor PCA

Stochastic gradient descent in high dimensions for multi-spiked tensor PCA

Large-Dimensional Positive Definite Covariance Estimation for High Frequency Data via Low-rank and Sparse Matrix Decomposition

Dynamical behavior of Oja PCA model for non-symmetric covariance matrix

Dynamic Principal Subspaces with Sparsity in High Dimensions

Biologically Plausible Online Principal Component Analysis Without Recurrent Neural Dynamics

Convergence analysis of a deterministic discrete time system of Oja's PCA learning algorithm.

Online Kernel Learning with a Near Optimal Sparsity Bound

Gradient-based sparse principal component analysis with extensions to online learning

On the discrete-time dynamics of a PCA learning algorithm

Phase Retrieval Using Iterative Projections: Dynamics in the Large Systems Limit

Dynamic Principal Subspaces in High Dimensions

Asymptotics of Empirical Eigen-structure for Ultra-high Dimensional Spiked Covariance Model

Diffusion approximations of Oja's online principal component analysis

Algorithmic thresholds for tensor PCA

Orthogonal Sparse PCA and Covariance Estimation via Procrustes Reformulation

Convergence of Oja's online principal component flow

Schrödinger PCA: On the Duality between Principal Component Analysis and Schrödinger Equation

Nearly optimal stochastic approximation for online principal subspace estimation