Abstract:In theoretical machine learning, the teacher–student paradigm is often employed as an effective metaphor for real-life tuition. A student network is trained on data generated by a fixed teacher network until it matches the instructor's ability to cope with the assigned task. The above scheme proves particularly relevant when the student network is overparameterized (namely, when larger layer sizes are employed) as compared to the underlying teacher network. Under these operating conditions, it is tempting to speculate that the student ability to handle the given task could be eventually stored in a sub-portion of the whole network. This latter should be to some extent reminiscent of the frozen teacher structure, according to suitable metrics, while being approximately invariant across different architectures of the student candidate network. Unfortunately, state-of-the-art conventional learning techniques could not help in identifying the existence of such an invariant subnetwork, due to the inherent degree of non-convexity that characterizes the examined problem. In this work, we take a decisive leap forward by proposing a radically different optimization scheme which builds on a spectral representation of the linear transfer of information between layers. The gradient is hence calculated with respect to both eigenvalues and eigenvectors with negligible increase in terms of computational and complexity load, as compared to standard training algorithms. Working in this framework, we could isolate a stable student substructure, that mirrors the true complexity of the teacher in terms of computing neurons, path distribution and topological attributes. When pruning unimportant nodes of the trained student, as follows a ranking that reflects the optimized eigenvalues, no degradation in the recorded performance is seen above a threshold that corresponds to the effective teacher size. The observed behavior can be pictured as a genuine second-order phase transition that bears universality traits. Code is available at: https://github.com/Jamba15/Spectral-regularization-teacher-student/tree/master.

Statistical Mechanics of Online Learning for Ensemble Teachers

On-line Learning of an Unlearnable True Teacher through Mobile Ensemble Teachers

Statistical Mechanics of On-line Ensemble Teacher Learning through a Novel Perceptron Learning Rule

Statistical Mechanics of On-line Learning When a Moving Teacher Goes around an Unlearnable True Teacher

Ensemble learning of linear perceptron; Online learning theory

Analysis of On-Line Learning when a Moving Teacher Goes around a True Teacher

Statistical Mechanics of Time Domain Ensemble Learning

Optimization of the Asymptotic Property of Mutual Learning Involving an Integration Mechanism of Ensemble Learning

Online Learning for the Random Feature Model in the Student-Teacher Framework

Dynamics of Meta-learning Representation in the Teacher-student Scenario

Leveraging Linear Independence of Component Classifiers: Optimizing Size and Prediction Accuracy for Online Ensembles

Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks

Spatial Ensemble: a Novel Model Smoothing Mechanism for Student-Teacher Framework

Learning performance in inverse Ising problems with sparse teacher couplings

Spatially heterogeneous learning by a deep student machine

How a student becomes a teacher: learning and forgetting through spectral methods

A Deterministic Analysis of an Online Convex Mixture of Expert Algorithms

Robust Modeling of Unknown Dynamical Systems via Ensemble Averaged Learning

The Copycat Perceptron: Smashing Barriers Through Collective Learning

On-line learning and generalisation in coupled perceptrons

On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting