Abstract:This paper investigates the intriguing question of whether we can create learning algorithms that automatically generate training data, learning environments, and curricula in order to help AI agents rapidly learn. We show that such algorithms are possible via Generative Teaching Networks (GTNs), a general approach that is, in theory, applicable to supervised, unsupervised, and reinforcement learning, although our experiments only focus on the supervised case. GTNs are deep neural networks that generate data and/or training environments that a learner (e.g. a freshly initialized neural network) trains on for a few SGD steps before being tested on a target task. We then differentiate through the entire learning process via meta-gradients to update the GTN parameters to improve performance on the target task. GTNs have the beneficial property that they can theoretically generate any type of data or training environment, making their potential impact large. This paper introduces GTNs, discusses their potential, and showcases that they can substantially accelerate learning. We also demonstrate a practical and exciting application of GTNs: accelerating the evaluation of candidate architectures for neural architecture search (NAS), which is rate-limited by such evaluations, enabling massive speed-ups in NAS. GTN-NAS improves the NAS state of the art, finding higher performing architectures when controlling for the search proposal mechanism. GTN-NAS also is competitive with the overall state of the art approaches, which achieve top performance while using orders of magnitude less computation than typical NAS methods. Speculating forward, GTNs may represent a first step toward the ambitious goal of algorithms that generate their own training data and, in doing so, open a variety of interesting new research questions and directions.

StarNet: Gradient-free Training of Deep Generative Models using Determined System of Linear Equations

Training Generative Adversarial Networks in One Stage

Do Deep Neural Network Solutions Form a Star Domain?

Training Generative Adversarial Networks by Solving Ordinary Differential Equations

Training Generative Adversarial Networks with Adaptive Composite Gradient

Generative Adversarial Nets

Local Convergence of Gradient Descent-Ascent for Training Generative Adversarial Networks

StarNet: Style-Aware 3D Point Cloud Generation

A Gradient-Guided Evolutionary Approach to Training Deep Neural Networks

Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data

Sketch-then-Edit Generative Adversarial Network

Idempotent Generative Network

A Gram-Gauss-Newton Method Learning Overparameterized Deep Neural Networks for Regression Problems.

Gradient Networks

Block-diagonal Hessian-free Optimization for Training Neural Networks

Genetic Algorithms for Evolving Deep Neural Networks

SCGN: novel generative model using the convergence of latent space by training

A Framework of Composite Functional Gradient Methods for Generative Adversarial Models

Gradient Descent based Optimization Algorithms for Deep Learning Models Training

Fed-DeepONet: Stochastic Gradient-Based Federated Training of Deep Operator Networks

Eigendecomposition-Free Training of Deep Networks for Linear Least-Square Problems