Abstract:Representations of the world environment play a crucial role in artificial intelligence. It is often inefficient to conduct reasoning and inference directly in the space of raw sensory representations, such as pixel values of images. Representation learning allows us to automatically discover suitable representations from raw sensory data. For example, given raw sensory data, a deep neural network learns nonlinear representations at its hidden layers, which are subsequently used for classification (or regression) at its output layer. This happens implicitly during training through minimizing a supervised or unsupervised loss in common practical regimes of deep learning, unlike the neural tangent kernel (NTK) regime. In this paper, we study the dynamics of such implicit nonlinear representation learning, which is beyond the NTK regime. We identify a pair of a new assumption and a novel condition, called the common model structure assumption and the data-architecture alignment condition. Under the common model structure assumption, the data-architecture alignment condition is shown to be sufficient for the global convergence and necessary for the global optimality. Moreover, our theory explains how and when increasing the network size does and does not improve the training behaviors in the practical regime. Our results provide practical guidance for designing a model structure: e.g., the common model structure assumption can be used as a justification for using a particular model structure instead of others. We also derive a new training framework based on the theory. The proposed framework is empirically shown to maintain competitive (practical) test performances while providing global convergence guarantees for deep residual neural networks with convolutions, skip connections, and batch normalization with standard benchmark datasets, including CIFAR-10, CIFAR-100, and SVHN.

Neural Eigenfunctions Are Structured Representation Learners

Self-Supervised Node Representation Learning Via Node-to-Neighbourhood Alignment.

Learning Structures for Deep Neural Networks

Learning Structured and Non-Redundant Representations with Deep Neural Networks

Deep Sketched Output Kernel Regression for Structured Prediction

Neural Representation of Shape-Dependent Laplacian Eigenfunctions

Reframing Neural Networks: Deep Structure in Overcomplete Representations

NeuralEF: Deconstructing Kernels by Deep Neural Networks.

Learning Structured Representations with Hyperbolic Embeddings

Stacked Convolutional Sparse Auto-Encoders for Representation Learning

Understanding Dynamics of Nonlinear Representation Learning and Its Application

Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps

Universality and Optimality of Structured Deep Kernel Networks

Embeddings and Representation Learning for Structured Data

Unsupervised representation learning with Hebbian synaptic and structural plasticity in brain-like feedforward neural networks

Towards Stable, Globally Expressive Graph Representations with Laplacian Eigenvectors

Eigen-GNN: a Graph Structure Preserving Plug-in for GNNs

Learning Deep Graph Representations via Convolutional Neural Networks

Universal structural patterns in sparse recurrent neural networks

Co-Neighbor Encoding Schema: A Light-cost Structure Encoding Method for Dynamic Link Prediction

Learning Neural Eigenfunctions for Unsupervised Semantic Segmentation