Abstract:In theoretical machine learning, the teacher–student paradigm is often employed as an effective metaphor for real-life tuition. A student network is trained on data generated by a fixed teacher network until it matches the instructor's ability to cope with the assigned task. The above scheme proves particularly relevant when the student network is overparameterized (namely, when larger layer sizes are employed) as compared to the underlying teacher network. Under these operating conditions, it is tempting to speculate that the student ability to handle the given task could be eventually stored in a sub-portion of the whole network. This latter should be to some extent reminiscent of the frozen teacher structure, according to suitable metrics, while being approximately invariant across different architectures of the student candidate network. Unfortunately, state-of-the-art conventional learning techniques could not help in identifying the existence of such an invariant subnetwork, due to the inherent degree of non-convexity that characterizes the examined problem. In this work, we take a decisive leap forward by proposing a radically different optimization scheme which builds on a spectral representation of the linear transfer of information between layers. The gradient is hence calculated with respect to both eigenvalues and eigenvectors with negligible increase in terms of computational and complexity load, as compared to standard training algorithms. Working in this framework, we could isolate a stable student substructure, that mirrors the true complexity of the teacher in terms of computing neurons, path distribution and topological attributes. When pruning unimportant nodes of the trained student, as follows a ranking that reflects the optimized eigenvalues, no degradation in the recorded performance is seen above a threshold that corresponds to the effective teacher size. The observed behavior can be pictured as a genuine second-order phase transition that bears universality traits. Code is available at: https://github.com/Jamba15/Spectral-regularization-teacher-student/tree/master.

Analysis of On-Line Learning when a Moving Teacher Goes around a True Teacher

Statistical Mechanics of On-line Learning When a Moving Teacher Goes around an Unlearnable True Teacher

On-line Learning of an Unlearnable True Teacher through Mobile Ensemble Teachers

Statistical Mechanics of Online Learning for Ensemble Teachers

Statistical Mechanics of On-line Ensemble Teacher Learning through a Novel Perceptron Learning Rule

Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks

Attacks on Online Learners: a Teacher-Student Analysis

Iterative Teacher-Aware Learning

Towards Black-box Iterative Machine Teaching

Learn to Teach: Improve Sample Efficiency in Teacher-student Learning for Sim-to-Real Transfer

Ensemble learning of linear perceptron; Online learning theory

On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting

Machine Teaching of Active Sequential Learners

How a student becomes a teacher: learning and forgetting through spectral methods

Understanding the Role of Adaptivity in Machine Teaching: The Case of Version Space Learners

Spatially heterogeneous learning by a deep student machine

Can a student learn optimally from two different teachers?

Dynamical and stationary properties of on-line learning from finite training sets

On-line learning and generalisation in coupled perceptrons

Learning to Teach with Dynamic Loss Functions