Abstract:Striking progress has been made in understanding cognition by analyzing how the brain is engaged in different modes of information processing. For instance, so-called synergistic information (information encoded by a set of neurons but not by any subset) plays a key role in areas of the human brain linked with complex cognition. However, two questions remain unanswered: (a) how and why a cognitive system can become highly synergistic; and (b) how informational states map onto artificial neural networks in various learning modes. Here we employ an information-decomposition framework to investigate neural networks performing cognitive tasks. Our results show that synergy increases as networks learn multiple diverse tasks, and that in tasks requiring integration of multiple sources, performance critically relies on synergistic neurons. Overall, our results suggest that synergy is used to combine information from multiple modalities—and more generally for flexible and efficient learning. These findings reveal new ways of investigating how and why learning systems employ specific information-processing strategies, and support the principle that the capacity for general-purpose learning critically relies on the system's information dynamics. What is the informational basis of learning in humans, animals, or, indeed, artificial neural networks (ANN)? Furthermore, how can these systems learn to solve multiple tasks simultaneously? These fundamental questions are, surprisingly, still not fully understood. One advantage of studying ANNs is that we can precisely probe learning-related changes. Here we draw on a recent branch of information theory, partial information decomposition, to examine how different types of information support different learning goals, and where, in ANNs. We show that adding noise to an ANN encourages it to keep copies of information at multiple nodes, promoting robustness. In contrast, whenever flexible learning is required, for instance when facing varied stimulus types or diverse tasks, individual neurons work together to represent information more abstractly. This work sheds light on how systems encode information differently according to their learning pressures, which can help us better understand how and why the human brain uses particular forms of information processing.

The Multi-Aggregation Process Neural Networks and Learning Algorithm

Process Neural Networks with Time-Varying Inputs and Outputs and Learning Algorithm

Dynamic Prediction Model Based on Process Neural Networks and Its Application

Multi-grain Cascade Recurrent Neural Network for Nonlinear Time-varying Process Soft Sensor Modeling

Some Theoretical Issues on Continuous Process Neural Networks

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Probabilistic Sequential Network for Deep Learning of Complex Process Data and Soft Sensor Application

Deep Spiking Neural Network Using Spatio-temporal Backpropagation with Variable Resistance.

Nonlinear Modeling of Neural Interaction for Spike Prediction Using the Staged Point-Process Model

Deep Neural Network-Embedded Stochastic Nonlinear State-Space Models and Their Applications to Process Monitoring

A SYMMETRIC TERNARY LOGIC BASED MULTI-VALUED LEARNING NETWORK

Training Algorithm of Process Neural Networks Based on Numerical Integration

Research and application of process neural network with two hidden-layer based on expansion of basis function

Deep Process Neural Network for Temporal Deep Learning

A Process Neural Network for Continuous Process Approximation and Its Application

Sequential Signal Mixing Aggregation for Message Passing Graph Neural Networks

Modeling spatio-temporal patterns of holistic functional brain networks via multi-head guided attention graph neural networks (Multi-Head GAGNNs)

Multi-Task Regression with Process Knowledge-Based Forest Learners in Process Industries.

Multi-graph aggregated graph neural network for heterogeneous graph representation learning

Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks