Abstract:In theoretical machine learning, the teacher–student paradigm is often employed as an effective metaphor for real-life tuition. A student network is trained on data generated by a fixed teacher network until it matches the instructor's ability to cope with the assigned task. The above scheme proves particularly relevant when the student network is overparameterized (namely, when larger layer sizes are employed) as compared to the underlying teacher network. Under these operating conditions, it is tempting to speculate that the student ability to handle the given task could be eventually stored in a sub-portion of the whole network. This latter should be to some extent reminiscent of the frozen teacher structure, according to suitable metrics, while being approximately invariant across different architectures of the student candidate network. Unfortunately, state-of-the-art conventional learning techniques could not help in identifying the existence of such an invariant subnetwork, due to the inherent degree of non-convexity that characterizes the examined problem. In this work, we take a decisive leap forward by proposing a radically different optimization scheme which builds on a spectral representation of the linear transfer of information between layers. The gradient is hence calculated with respect to both eigenvalues and eigenvectors with negligible increase in terms of computational and complexity load, as compared to standard training algorithms. Working in this framework, we could isolate a stable student substructure, that mirrors the true complexity of the teacher in terms of computing neurons, path distribution and topological attributes. When pruning unimportant nodes of the trained student, as follows a ranking that reflects the optimized eigenvalues, no degradation in the recorded performance is seen above a threshold that corresponds to the effective teacher size. The observed behavior can be pictured as a genuine second-order phase transition that bears universality traits. Code is available at: https://github.com/Jamba15/Spectral-regularization-teacher-student/tree/master.

Free Probability for predicting the performance of feed-forward fully connected neural networks

Relative gradient optimization of the Jacobian term in unsupervised deep learning

Estimating the Jacobian matrix of an unknown multivariate function from sample values by means of a neural network

Understanding Gradient Descent through the Training Jacobian

Training on the Edge of Stability Is Caused by Layerwise Jacobian Alignment

Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians

Stationary Points of a Shallow Neural Network with Quadratic Activations and the Global Optimality of the Gradient Descent Algorithm

Gradient-Free Neural Network Training on the Edge

Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods

How a student becomes a teacher: learning and forgetting through spectral methods

Empirical Analysis of the Hessian of Over-Parametrized Neural Networks

A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networks

Spectrum concentration in deep residual learning: a free probability approach

A Fine-Grained Spectral Perspective on Neural Networks

On the saddle point problem for non-convex optimization

The Hessian by blocks for neural network by backward propagation

High dimensional analysis reveals conservative sharpening and a stochastic edge of stability

A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities

On the ISS Property of the Gradient Flow for Single Hidden-Layer Neural Networks with Linear Activations

Gradient Sketches for Training Data Attribution and Studying the Loss Landscape

Can Stability be Detrimental? Better Generalization through Gradient Descent Instabilities