Abstract:The brain is not only constrained by energy needed to fuel computation, but it is also constrained by energy needed to form memories. Experiments have shown that learning simple conditioning tasks which might require only a few synaptic updates, already carries a significant metabolic cost. Yet, learning a task like MNIST to 95% accuracy appears to require at least 10 8 synaptic updates. Therefore the brain has likely evolved to be able to learn using as little energy as possible. We explored the energy required for learning in feedforward neural networks. Based on a parsimonious energy model, we propose two plasticity restricting algorithms that save energy: 1) only modify synapses with large updates, and 2) restrict plasticity to subsets of synapses that form a path through the network. In biology networks are often much larger than the task requires, yet vanilla backprop prescribes to update all synapses. In particular in this case, large savings can be achieved while only incurring a slightly worse learning time. Thus competitively restricting plasticity helps to save metabolic energy associated to synaptic plasticity. The results might lead to a better understanding of biological plasticity and a better match between artificial and biological learning. Moreover, the algorithms might benefit hardware because also electronic memory storage is energetically costly. There is increasing evidence learning already very simple tasks in animals requires substantial amounts of metabolic energy. This raises the question how much energy it costs to learn more complex tasks. For instance, the well known backpropagation algorithm always updates all synapses in the network and one can wonder if that is necessary. In the theoretical study we estimate the energy needed for an artificial neural network to learn to classify the well-known MNIST data set. We find that in particular in larger networks, substantial energy savings can be achieved, by carefully selecting which synapses to update. This is particular relevant for the neural networks in the brain that often contain millions of neurons. This study will hopefully lead to a better understanding of learning processes in biology. Moreover, as training large artificial neural networks consumes substantial amount of electric energy, the savings algorithms proposed here, might help to reduce those costs as well.

Competitive plasticity to reduce the energetic costs of learning

Competitive plasticity to reduce the energetic costs of learning

Overcoming Long-Term Catastrophic Forgetting Through Adversarial Neural Pruning and Synaptic Consolidation

Lazy learning: a biologically-inspired plasticity rule for fast and energy efficient synaptic plasticity

Energetics of stochastic BCM type synaptic plasticity and storing of accurate information

Optimality of short-term synaptic plasticity in modelling certain dynamic environments

Metabolic constraints on synaptic learning and memory

Balancing complexity, performance and plausibility to meta learn plasticity rules in recurrent spiking networks.

Realizing Linear Synaptic Plasticity in Electric Double Layer-Gated Transistors for Improved Predictive Accuracy and Efficiency in Neuromorphic Computing

Toward Efficient Processing and Learning with Spikes: New Approaches for Multispike Learning

Evolving Decomposed Plasticity Rules for Information-Bottlenecked Meta-Learning

Contrasting Advantages of Learning With Random Weights and Backpropagation in Non-Volatile Memory Neural Networks

Reinforcement learning when your life depends on it: A neuro-economic theory of learning

Sparse coding with a somato-dendritic rule

Improving energy efficiency and classification accuracy of neuromorphic chips by learning binary synaptic crossbars

The brain as an efficient and robust adaptive learner

Using local plasticity rules to train recurrent neural networks

Neuron-centric Hebbian Learning

A brain-inspired algorithm that mitigates catastrophic forgetting of artificial and spiking neural networks with low computational cost

Differentiable plasticity: training plastic neural networks with backpropagation

Efficient dendritic learning as an alternative to synaptic plasticity hypothesis