Abstract:Recent years have been marked with the fast-pace diversification and increasing ubiquity of machine learning applications. Yet, a firm theoretical understanding of the surprising efficiency of neural networks to learn from high-dimensional data still proves largely elusive. In this endeavour, analyses inspired by statistical physics have proven instrumental, enabling the tight asymptotic characterization of the learning of neural networks in high dimensions, for a broad class of solvable models. This manuscript reviews the tools and ideas underlying recent progress in this line of work. We introduce a generic model -- the sequence multi-index model -- which encompasses numerous previously studied models as special instances. This unified framework covers a broad class of machine learning architectures with a finite number of hidden units, including multi-layer perceptrons, autoencoders, attention mechanisms; and tasks, including (un)supervised learning, denoising, contrastive learning, in the limit of large data dimension, and comparably large number of samples. We explicate in full detail the analysis of the learning of sequence multi-index models, using statistical physics techniques such as the replica method and approximate message-passing algorithms. This manuscript thus provides a unified presentation of analyses reported in several previous works, and a detailed overview of central techniques in the field of statistical physics of machine learning. This review should be a useful primer for machine learning theoreticians curious of statistical physics approaches; it should also be of value to statistical physicists interested in the transfer of such ideas to the study of neural networks.

High-Dimensional Non-Convex Landscapes and Gradient Descent Dynamics

High-dimensional non-convex landscapes and gradient descent dynamics

Emergent properties of the local geometry of neural loss landscapes

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

A gentle introduction to gradient-based optimization and variational inequalities for machine learning

From Zero to Hero: How local curvature at artless initial conditions leads away from bad minima

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index models

Information Geometry of Dynamics on Graphs and Hypergraphs

Landscapes of Non-gradient Dynamics Without Detailed Balance: Stable Limit Cycles and Multiple Attractors

High-dimensional learning of narrow neural networks

Gradient descent in Gaussian random fields as a toy model for high-dimensional optimisation in deep learning

A Geometric Understanding of Natural Gradient

Extended convexity and smoothness and their applications in deep learning

Differential geometry and stochastic dynamics with deep learning numerics

Thresholds of descending algorithms in inference problems

Data-Driven Computing Methods for Nonlinear Physics Systems with Geometric Constraints

Kernels, data & physics

A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networks

Black holes and the loss landscape in machine learning

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization