Abstract:In a physical neural system, where storage and processing are intimately intertwined, the rules for adjusting the synaptic weights can only depend on variables that are available locally, such as the activity of the pre- and post-synaptic neurons, resulting in local learning rules. A systematic framework for studying the space of local learning rules is obtained by first specifying the nature of the local variables, and then the functional form that ties them together into each learning rule. Such a framework enables also the systematic discovery of new learning rules and exploration of relationships between learning rules and group symmetries. We study polynomial local learning rules stratified by their degree and analyze their behavior and capabilities in both linear and non-linear units and networks. Stacking local learning rules in deep feedforward networks leads to deep local learning. While deep local learning can learn interesting representations, it cannot learn complex input-output functions, even when targets are available for the top layer. Learning complex input-output functions requires local deep learning where target information is communicated to the deep layers through a backward learning channel. The nature of the communicated information about the targets and the structure of the learning channel partition the space of learning algorithms. We estimate the learning channel capacity associated with several algorithms and show that backpropagation outperforms them by simultaneously maximizing the information rate and minimizing the computational cost, even in recurrent networks. The theory clarifies the concept of Hebbian learning, establishes the power and limitations of local learning rules, introduces the learning channel which enables a formal analysis of the optimality of backpropagation, and explains the sparsity of the space of learning rules discovered so far.

A biologically plausible neural network for local supervision in cortical microcircuits

A simple normative network approximates local non-Hebbian learning in the cortex

Learning efficient backprojections across cortical hierarchies in real time

A More Biologically Plausible Local Learning Rule for ANNs

Towards Biologically Plausible Computing: A Comprehensive Comparison

Approximating Back-propagation for a Biologically Plausible Local Learning Rule in Spiking Neural Networks

Dendritic cortical microcircuits approximate the backpropagation algorithm

PC-SNN: Supervised Learning with Local Hebbian Synaptic Plasticity Based on Predictive Coding in Spiking Neural Networks

Towards New Generation, Biologically Plausible Deep Neural Network Learning

Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain

A Biologically Plausible Neural Network for Multichannel Canonical Correlation Analysis

Biologically Inspired Structure Learning with Reverse Knowledge Distillation for Spiking Neural Networks

Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry

Learning arbitrary dynamics in efficient, balanced spiking networks using local plasticity rules

Using local plasticity rules to train recurrent neural networks

Local lateral connectivity is sufficient for replicating cortex-like topographical organization in deep neural networks

Exploring the Architectural Biases of the Canonical Cortical Microcircuit

BioLeaF: A Bio-plausible Learning Framework for Training of Spiking Neural Networks

A Theory of Local Learning, the Learning Channel, and the Optimality of Backpropagation

Towards Biologically Plausible Deep Learning