Abstract:Data augmentation is a widely used technique and an essential ingredient in the recent advance in self-supervised representation learning. By preserving the similarity between augmented data, the resulting data representation can improve various downstream analyses and achieve state-of-the-art performance in many applications. Despite the empirical effectiveness, most existing methods lack theoretical understanding under a general nonlinear setting. To fill this gap, we develop a statistical framework on a low-dimension product manifold to model the data augmentation transformation. Under this framework, we introduce a new representation learning method called augmentation invariant manifold learning and design a computationally efficient algorithm by reformulating it as a stochastic optimization problem. Compared with existing self-supervised methods, the new method simultaneously exploits the manifold's geometric structure and invariant property of augmented data and has an explicit theoretical guarantee. Our theoretical investigation characterizes the role of data augmentation in the proposed method and reveals why and how the data representation learned from augmented data can improve the $k$-nearest neighbor classifier in the downstream analysis, showing that a more complex data augmentation leads to more improvement in downstream analysis. Finally, numerical experiments on simulated and real datasets are presented to demonstrate the merit of the proposed method.

On order determination by predictor augmentation

Combining eigenvalues and variation of eigenvectors for order determination

Order Determination for Spiked Models

Augmented Estimation of Principal Component Subspace in High Dimensions

Determine the number of clusters by data augmentation

Large-Dimensional Positive Definite Covariance Estimation for High Frequency Data via Low-rank and Sparse Matrix Decomposition

A new sufficient dimension reduction method via rank divergence

Testing Predictor Significance with Ultra High Dimensional Multivariate Responses

Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions.

ON SUPERVISED REDUCTION AND ITS DUAL

Exploring Dimension Learning Via a Penalized Probabilistic Principal Component Analysis

Dimension Reduction in Contextual Online Learning Via Nonparametric Variable Selection

High Dimensional Factor Analysis with Weak Factors

Functional sufficient dimension reduction through distance covariance

Prescriptive PCA: Dimensionality Reduction for Two-stage Stochastic Optimization

Augmentation Invariant Manifold Learning

A Principal Square Response Forward Regression Method for Dimension Reduction

Dimensionality Determination: A Thresholding Double Ridge Ratio Approach

A random-perturbation-based rank estimator of the number of factors

ON THE SECOND-ORDER INVERSE REGRESSION METHODS FOR A GENERAL TYPE OF ELLIPTICAL PREDICTORS

Rank Reduction Autoencoders -- Enhancing interpolation on nonlinear manifolds