Abstract:Representations of the world environment play a crucial role in artificial intelligence. It is often inefficient to conduct reasoning and inference directly in the space of raw sensory representations, such as pixel values of images. Representation learning allows us to automatically discover suitable representations from raw sensory data. For example, given raw sensory data, a deep neural network learns nonlinear representations at its hidden layers, which are subsequently used for classification (or regression) at its output layer. This happens implicitly during training through minimizing a supervised or unsupervised loss in common practical regimes of deep learning, unlike the neural tangent kernel (NTK) regime. In this paper, we study the dynamics of such implicit nonlinear representation learning, which is beyond the NTK regime. We identify a pair of a new assumption and a novel condition, called the common model structure assumption and the data-architecture alignment condition. Under the common model structure assumption, the data-architecture alignment condition is shown to be sufficient for the global convergence and necessary for the global optimality. Moreover, our theory explains how and when increasing the network size does and does not improve the training behaviors in the practical regime. Our results provide practical guidance for designing a model structure: e.g., the common model structure assumption can be used as a justification for using a particular model structure instead of others. We also derive a new training framework based on the theory. The proposed framework is empirically shown to maintain competitive (practical) test performances while providing global convergence guarantees for deep residual neural networks with convolutions, skip connections, and batch normalization with standard benchmark datasets, including CIFAR-10, CIFAR-100, and SVHN.

To What Extent Do Different Neural Networks Learn the Same Representation: A Neuron Activation Subspace Match Approach

Towards Understanding Learning Representations: to What Extent Do Different Neural Networks Learn the Same Representation.

Convergent Learning: Do different neural networks learn the same representations?

Bridging the Semantic Latent Space Between Brain and Machine: Similarity is All You Need

Similarity of Neural Network Representations Revisited

Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?

Similarity of Neural Network Models: A Survey of Functional and Representational Measures

Deep Co-Attention Network for Multi-View Subspace Learning

On the similarities of representations in artificial and brain neural networks for speech recognition

You Only Learn One Representation: Unified Network for Multiple Tasks

Universality of representation in biological and artificial neural networks

When Representations Align: Universality in Representation Learning Dynamics

Decoupling Semantic Similarity from Spatial Alignment for Neural Networks

Identifying Sub-networks in Neural Networks via Functionally Similar Representations

Formation of Representations in Neural Networks

Graph-Based Similarity of Neural Network Representations.

Learning Network Representations with Neighboring Affinity

Neural Similarity Learning

Understanding Dynamics of Nonlinear Representation Learning and Its Application

Network Representation Learning Based on Social Similarities

Evaluating (and Improving) the Correspondence Between Deep Neural Networks and Human Representations