Abstract:Representations of the world environment play a crucial role in artificial intelligence. It is often inefficient to conduct reasoning and inference directly in the space of raw sensory representations, such as pixel values of images. Representation learning allows us to automatically discover suitable representations from raw sensory data. For example, given raw sensory data, a deep neural network learns nonlinear representations at its hidden layers, which are subsequently used for classification (or regression) at its output layer. This happens implicitly during training through minimizing a supervised or unsupervised loss in common practical regimes of deep learning, unlike the neural tangent kernel (NTK) regime. In this paper, we study the dynamics of such implicit nonlinear representation learning, which is beyond the NTK regime. We identify a pair of a new assumption and a novel condition, called the common model structure assumption and the data-architecture alignment condition. Under the common model structure assumption, the data-architecture alignment condition is shown to be sufficient for the global convergence and necessary for the global optimality. Moreover, our theory explains how and when increasing the network size does and does not improve the training behaviors in the practical regime. Our results provide practical guidance for designing a model structure: e.g., the common model structure assumption can be used as a justification for using a particular model structure instead of others. We also derive a new training framework based on the theory. The proposed framework is empirically shown to maintain competitive (practical) test performances while providing global convergence guarantees for deep residual neural networks with convolutions, skip connections, and batch normalization with standard benchmark datasets, including CIFAR-10, CIFAR-100, and SVHN.

The Structure Transfer Machine Theory and Applications

Learning Exemplar-Represented Manifolds in Latent Space for Classification.

Deep Clustering and Representation Learning that Preserves Geometric Structures

Learning Structure-Aware Transformations for Arbitrary Image Style Transfer

Extreme Learning Machine Via Free Sparse Transfer Representation Optimization

Deep Manifold Transformation for Nonlinear Dimensionality Reduction

Point Cloud Domain Adaptation Via Masked Local 3D Structure Prediction

Structure-preserving visualization for single-cell RNA-Seq profiles using deep manifold transformation with batch-correction

Preserving Structural Consistency in Arbitrary Artist and Artwork Style Transfer

Understanding How Feature Structure Transfers in Transfer Learning.

Understanding Dynamics of Nonlinear Representation Learning and Its Application

Flexible deep transfer learning by separate feature embeddings and manifold alignment

Matrix Tri-Factorization With Manifold Regularizations For Zero-Shot Learning

Structure Evolution on Manifold for Graph Learning

Deep Manifold Transformation for Protein Representation Learning

Transfer Sparse Machine: Matching Joint Distribution By Subspace Learning And Classifier Transduction

Constrained Deep Transfer Feature Learning and its Applications

Manifold Transfer Learning via Discriminant Regression Analysis

Representation Learning on Large and Small Data

Learning explicitly transferable representations for domain adaptation

Unsupervisedly Learned Latent Graphs as Transferable Representations