Abstract:Transformers have revolutionized machine learning models of language and vision, but their connection with neuroscience remains tenuous. Built from attention layers, they require a mass comparison of queries and keys that is difficult to perform using traditional neural circuits. Here, we show that neurons can implement attention-like computations using short-term, Hebbian synaptic potentiation. We call our mechanism the match-and-control principle and it proposes that when activity in an axon is synchronous, or matched, with the somatic activity of a neuron that it synapses onto, the synapse can be briefly strongly potentiated, allowing the axon to take over, or control, the activity of the downstream neuron for a short time. In our scheme, the keys and queries are represented as spike trains and comparisons between the two are performed in individual spines allowing for hundreds of key comparisons per query and roughly as many keys and queries as there are neurons in the network. Many of the most impressive recent advances in machine learning, from generating images from text to human-like chatbots, are based on a neural network architecture known as the transformer. Transformers are built from so-called attention layers which perform large numbers of comparisons between the vector outputs of the previous layers, allowing information to flow through the network in a more dynamic way than previous designs. This large number of comparisons is computationally expensive and has no known analogue in the brain. Here, we show that a variation on a learning mechanism familiar in neuroscience, Hebbian learning, can implement a transformer-like attention computation if the synaptic weight changes are large and rapidly induced. We call our method the match-and-control principle and it proposes that when presynaptic and postsynaptic spike trains match up, small groups of synapses can be transiently potentiated allowing a few presynaptic axons to control the activity of a neuron. To demonstrate the principle, we build a model of a pyramidal neuron and use it to illustrate the power and limitations of the idea.

The Attention Mechanism Demystiûed

AttentionViz: A Global View of Transformer Attention

Generalized Probabilistic Attention Mechanism in Transformers

Attention is All you Need

Attention Mechanism, Transformers, BERT, and GPT: Tutorial and Survey

Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains

From Cognition to Computation: A Comparative Review of Human Attention and Transformer Architectures

The Asymptotic Behavior of Attention in Transformers

Rethinking the role of attention mechanism: a causality perspective

Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers

Short-term Hebbian learning can implement transformer-like attention

What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis

Attention Is Not All You Need Anymore

Attention as an RNN

Transformer Mechanisms Mimic Frontostriatal Gating Operations When Trained on Human Working Memory Tasks

Attention as a Hypernetwork

Attention: Marginal Probability is All You Need?

The Quarks of Attention

A Primal-Dual Framework for Transformers and Neural Networks

An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models