Abstract:Deep neural feedforward networks are effective models for a wide array of problems, but training and deploying such networks presents a significant energy cost. Spiking neural networks (SNNs), which are modeled after biologically realistic neurons, offer a potential solution when deployed correctly on neuromorphic computing hardware. Still, many applications train SNNs offline, and running network training directly on neuromorphic hardware is an ongoing research problem. The primary hurdle is that back-propagation, which makes training such artificial deep networks possible, is biologically implausible. Neuroscientists are uncertain about how the brain would propagate a precise error signal backward through a network of neurons. Recent progress addresses part of this question, e.g., the weight transport problem, but a complete solution remains intangible. In contrast, novel learning rules based on the information bottleneck (IB) train each layer of a network independently, circumventing the need to propagate errors across layers. Instead, propagation is implicit due the layers' feedforward connectivity. These rules take the form of a three-factor Hebbian update a global error signal modulates local synaptic updates within each layer. Unfortunately, the global signal for a given layer requires processing multiple samples concurrently, and the brain only sees a single sample at a time. We propose a new three-factor update rule where the global signal correctly captures information across samples via an auxiliary memory network. The auxiliary network can be trained a priori independently of the dataset being used with the primary network. We demonstrate comparable performance to baselines on image classification tasks. Interestingly, unlike back-propagation-like schemes where there is no link between learning and memory, our rule presents a direct connection between working memory and synaptic updates. To the best of our knowledge, this is the first rule to make this link explicit. We explore these implications in initial experiments examining the effect of memory capacity on learning performance. Moving forward, this work suggests an alternate view of learning where each layer balances memory-informed compression against task performance. This view naturally encompasses several key aspects of neural computation, including memory, efficiency, and locality.

Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?

Normative framework for deriving neural networks with multi-compartmental neurons and non-Hebbian plasticity

Duality Principle and Biologically Plausible Learning: Connecting the Representer Theorem and Hebbian Learning

Differentiable Optimization of Similarity Scores Between Models and Brains

A Normative Theory of Adaptive Dimensionality Reduction in Neural Networks

To What Extent Do Different Neural Networks Learn the Same Representation: A Neuron Activation Subspace Match Approach

A simple normative network approximates local non-Hebbian learning in the cortex

Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry

Bridging the Semantic Latent Space Between Brain and Machine: Similarity is All You Need

Neuron-centric Hebbian Learning

Modeling Hebb Learning Rule for Unsupervised Learning.

Towards Understanding Learning Representations: to What Extent Do Different Neural Networks Learn the Same Representation.

What should a neuron aim for? Designing local objective functions based on information theory

Distributed Memory and Localized Memory in Neural Networks

Strong anti-Hebbian plasticity alters the convexity of network attractor landscapes

Unlocking the Potential of Similarity Matching: Scalability, Supervision and Pre-training

Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks

Layerwise Hebbian/anti-Hebbian (HaH) Learning In Deep Networks: A Neuro-inspired Approach To Robustness

Unsupervised learning by a nonlinear network with Hebbian excitatory and anti-Hebbian inhibitory neurons

Hebbian Descent: A Unified View on Log-Likelihood Learning

Information bottleneck-based Hebbian learning rule naturally ties working memory and synaptic updates