Abstract:The brain is not only constrained by energy needed to fuel computation, but it is also constrained by energy needed to form memories. Experiments have shown that learning simple conditioning tasks which might require only a few synaptic updates, already carries a significant metabolic cost. Yet, learning a task like MNIST to 95% accuracy appears to require at least 10 8 synaptic updates. Therefore the brain has likely evolved to be able to learn using as little energy as possible. We explored the energy required for learning in feedforward neural networks. Based on a parsimonious energy model, we propose two plasticity restricting algorithms that save energy: 1) only modify synapses with large updates, and 2) restrict plasticity to subsets of synapses that form a path through the network. In biology networks are often much larger than the task requires, yet vanilla backprop prescribes to update all synapses. In particular in this case, large savings can be achieved while only incurring a slightly worse learning time. Thus competitively restricting plasticity helps to save metabolic energy associated to synaptic plasticity. The results might lead to a better understanding of biological plasticity and a better match between artificial and biological learning. Moreover, the algorithms might benefit hardware because also electronic memory storage is energetically costly. There is increasing evidence learning already very simple tasks in animals requires substantial amounts of metabolic energy. This raises the question how much energy it costs to learn more complex tasks. For instance, the well known backpropagation algorithm always updates all synapses in the network and one can wonder if that is necessary. In the theoretical study we estimate the energy needed for an artificial neural network to learn to classify the well-known MNIST data set. We find that in particular in larger networks, substantial energy savings can be achieved, by carefully selecting which synapses to update. This is particular relevant for the neural networks in the brain that often contain millions of neurons. This study will hopefully lead to a better understanding of learning processes in biology. Moreover, as training large artificial neural networks consumes substantial amount of electric energy, the savings algorithms proposed here, might help to reduce those costs as well.

Metabolize Neural Network

Neural Metamorphosis

A Novel Paradigm for Neural Computation: X-Net with Learnable Neurons and Adaptable Structure

Constraints of Metabolic Energy on the Number of Synaptic Connections of Neurons and the Density of Neuronal Networks

Neuromodulated Meta-Learning

Unveiling the role of local metabolic constraints on the structure and activity of spiking neural networks

A Novel Neuron Connection Model Mimicking Human Beings.

Exploration Of A Mechanism To Form Bionic, Self-Growing And Self-Organizing Neural Network

Competitive plasticity to reduce the energetic costs of learning

METABOLIC COST AS AN ORGANIZING PRINCIPLE FOR COOPERATIVE LEARNING

Continual Learning with Deep Artificial Neurons

Under the Hood of Neural Networks: Characterizing Learned Representations by Functional Neuron Populations and Network Ablations

Graph Metanetworks for Processing Diverse Neural Architectures

Network properties determine neural network performance

SynapsNet: Enhancing Neuronal Population Dynamics Modeling via Learning Functional Connectivity

Testing Neural Network Classifiers Based on Metamorphic Relations

A deep language model to predict metabolic network equilibria

Stable Network Morphism

Efficient Neural Networks Algorithm Based on Connectivity Properties of Biological Brain Networks

Neural Networks from Biological to Artificial and Vice Versa

Machine Learning Using Neural Networks for Metabolomic Pathway Analyses