Abstract:Striking progress has been made in understanding cognition by analyzing how the brain is engaged in different modes of information processing. For instance, so-called synergistic information (information encoded by a set of neurons but not by any subset) plays a key role in areas of the human brain linked with complex cognition. However, two questions remain unanswered: (a) how and why a cognitive system can become highly synergistic; and (b) how informational states map onto artificial neural networks in various learning modes. Here we employ an information-decomposition framework to investigate neural networks performing cognitive tasks. Our results show that synergy increases as networks learn multiple diverse tasks, and that in tasks requiring integration of multiple sources, performance critically relies on synergistic neurons. Overall, our results suggest that synergy is used to combine information from multiple modalities—and more generally for flexible and efficient learning. These findings reveal new ways of investigating how and why learning systems employ specific information-processing strategies, and support the principle that the capacity for general-purpose learning critically relies on the system's information dynamics. What is the informational basis of learning in humans, animals, or, indeed, artificial neural networks (ANN)? Furthermore, how can these systems learn to solve multiple tasks simultaneously? These fundamental questions are, surprisingly, still not fully understood. One advantage of studying ANNs is that we can precisely probe learning-related changes. Here we draw on a recent branch of information theory, partial information decomposition, to examine how different types of information support different learning goals, and where, in ANNs. We show that adding noise to an ANN encourages it to keep copies of information at multiple nodes, promoting robustness. In contrast, whenever flexible learning is required, for instance when facing varied stimulus types or diverse tasks, individual neurons work together to represent information more abstractly. This work sheds light on how systems encode information differently according to their learning pressures, which can help us better understand how and why the human brain uses particular forms of information processing.

Collective computational properties of neural networks: New learning mechanisms

Demolition and Reinforcement of Memories in Spin-Glass-like Neural Networks

Dynamic Properties and a New Learning Mechanism in Higher Order Neural Networks

The brain as an efficient and robust adaptive learner

A Novel Neural Network for Associative Memory Via Dynamical Systems

Learning and Structure of Neuronal Networks

New elements for a network (including brain) general theory during learning period

Magnetic Tunnel Junction Based Long-Term Short-Term Stochastic Synapse for a Spiking Neural Network with On-Chip STDP Learning

Relative Entropy and Learning Rules.

Dreaming neural networks: Forgetting spurious memories and reinforcing pure ones

Entanglement transition in deep neural quantum states

Deep neural networks for direct, featureless learning through observation: the case of 2d spin models

Unsupervised and Supervised learning by Dense Associative Memory under replica symmetry breaking

Efficient supervised learning in networks with binary synapses

Large Associative Memory Problem in Neurobiology and Machine Learning

Neural Networks as Spin Models: From Glass to Hidden Order Through Training

Probabilistic models, compressible interactions, and neural coding

Beyond spiking networks: The computational advantages of dendritic amplification and input segregation

Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks

Learning, Storing, and Disentangling Correlated Patterns in Neural Networks.

Statistical mechanics for networks of real neurons