Abstract:A key driver of mammalian intelligence is the ability to represent incoming sensory information across multiple abstraction levels. For example, in the visual ventral stream, incoming signals are first represented as low-level edge filters and then transformed into high-level object representations. Similar hierarchical structures routinely emerge in artificial neural networks (ANNs) trained for object recognition tasks, suggesting that similar structures may underlie biological neural networks. However, the classical ANN training algorithm, backpropagation, is considered biologically implausible, and thus alternative biologically plausible training methods have been developed such as Equilibrium Propagation, Deep Feedback Control, Supervised Predictive Coding, and Dendritic Error Backpropagation. Several of those models propose that local errors are calculated for each neuron by comparing apical and somatic activities. Notwithstanding, from a neuroscience perspective, it is not clear how a neuron could compare compartmental signals. Here, we propose a solution to this problem in that we let the apical feedback signal change the postsynaptic firing rate and combine this with a differential Hebbian update, a rate-based version of classical spiking time-dependent plasticity (STDP). We prove that weight updates of this form minimize two alternative loss functions that we prove to be equivalent to the error-based losses used in machine learning: the inference latency and the amount of top-down feedback necessary. Moreover, we show that the use of differential Hebbian updates works similarly well in other feedback-based deep learning frameworks such as Predictive Coding or Equilibrium Propagation. Finally, our work removes a key requirement of biologically plausible models for deep learning and proposes a learning mechanism that would explain how temporal Hebbian learning rules can implement supervised hierarchical learning.

Learning Sparse, Distributed Representations using the Hebbian Principle

Unsupervised representation learning with Hebbian synaptic and structural plasticity in brain-like feedforward neural networks

Layerwise Hebbian/anti-Hebbian (HaH) Learning In Deep Networks: A Neuro-inspired Approach To Robustness

Competitive learning to generate sparse representations for associative memory

Hebbian Learning from First Principles

Learning cortical hierarchies with temporal Hebbian updates

Can a Hebbian-like learning rule be avoiding the curse of dimensionality in sparse distributed data?

Representation Learning in a Decomposed Encoder Design for Bio-inspired Hebbian Learning

Neuro-Inspired Deep Neural Networks with Sparse, Strong Activations

Superposed Episodic and Semantic Memory via Sparse Distributed Representation

Learning Word Representations with Hierarchical Sparse Coding

Training Convolutional Neural Networks With Hebbian Principal Component Analysis

Spiking representation learning for associative memories

Deviant Learning Algorithm: Learning Sparse Mismatch Representations through Time and Space

Duality Principle and Biologically Plausible Learning: Connecting the Representer Theorem and Hebbian Learning

Clustering Inductive Biases with Unrolled Networks

Dense Hebbian neural networks: a replica symmetric picture of unsupervised learning

The Relative Advantages of Sparse Versus Distributed Encoding for Associative Neuronal Networks in the Brain

Representation Memorization for Fast Learning New Knowledge without Forgetting

Sparse, Geometric Autoencoder Models of V1