Abstract:In the era of big data, reducing data dimensionality is critical in many areas of science. Widely used Principal Component Analysis (PCA) addresses this problem by computing a low dimensional data embedding that maximally explain variance of the data. However, PCA has two major weaknesses. Firstly, it only considers linear correlations among variables (features), and secondly it is not suitable for categorical data. We resolve these issues by proposing Maximally Correlated Principal Component Analysis (MCPCA). MCPCA computes transformations of variables whose covariance matrix has the largest Ky Fan norm. Variable transformations are unknown, can be nonlinear and are computed in an optimization. MCPCA can also be viewed as a multivariate extension of Maximal Correlation. For jointly Gaussian variables we show that the covariance matrix corresponding to the identity (or the negative of the identity) transformations majorizes covariance matrices of non-identity functions. Using this result we characterize global MCPCA optimizers for nonlinear functions of jointly Gaussian variables for every rank constraint. For categorical variables we characterize global MCPCA optimizers for the rank one constraint based on the leading eigenvector of a matrix computed using pairwise joint distributions. For a general rank constraint we propose a block coordinate descend algorithm and show its convergence to stationary points of the MCPCA optimization. We compare MCPCA with PCA and other state-of-the-art dimensionality reduction methods including Isomap, LLE, multilayer autoencoders (neural networks), kernel PCA, probabilistic PCA and diffusion maps on several synthetic and real datasets. We show that MCPCA consistently provides improved performance compared to other methods.

Covariance and PCA for Categorical Variables

Robust Principal Component Analysis Based on Maximum Correntropy Criterion

Functional PCA With Covariate-Dependent Mean and Covariance Structure

Maximally Correlated Principal Component Analysis

A Covariance-Free Iterative Principal Component Analysis for High Dimensional and Large Scale Data

Covariance descriptors on a Gaussian manifold and their application to image set classification

Robust Covariance Estimation for Distributed Principal Component Analysis

Covariance Discriminative Learning: A Natural and Efficient Approach to Image Set Classification

Generalized probabilistic principal component analysis of correlated data

Principal Components Analysis in Clinical Studies

CAVIAR: Categorical-Variable Embeddings for Accurate and Robust Inference

On the properties of distance covariance for categorical data: Robustness, sure screening, and approximate null distributions

Diagonally-Dominant Principal Component Analysis

Two derivations of Principal Component Analysis on datasets of distributions

Independent Component Analysis via Distance Covariance

More About Covariance Descriptors for Image Set Coding: Log-Euclidean Framework based Kernel Matrix Representation

Sparse principal component analysis via regularized low rank matrix approximation

Principal Component Analysis in Space Forms

Sparse Factor Analysis for Categorical Data with the Group-Sparse Generalized Singular Value Decomposition

Visualizing class specific heterogeneous tendencies in categorical data