Abstract:This work proposes a new perspective on the structure of Neural Networks (NNs). Traditional Neural Networks are typically tree-like structures for convenience, which can be predefined or learned by NAS methods. However, such a structure can not facilitate communications between nodes at the same level or signal transmissions to previous levels. These defects prevent effective collaboration, restricting the capabilities of neural networks. It is well-acknowledged that the biological neural system contains billions of neural units. Their connections are far more complicated than the current NN structure. To enhance the representational ability of neural networks, existing works try to increase the depth of the neural network and introduce more parameters. However, they all have limitations with constrained parameters. In this work, we introduce a synchronous graph-based structure to establish a novel way of organizing the neural units: the Neural Modules. This framework allows any nodes to communicate with each other and encourages neural units to work collectively, demonstrating a departure from the conventional constrained paradigm. Such a structure also provides more candidates for the NAS methods. Furthermore, we also propose an elegant regularization method to organize neural units into multiple independent, balanced neural modules systematically. This would be convenient for handling these neural modules in parallel. Compared to traditional NNs, our method unlocks the potential of NNs from tree-like structures to general graphs and makes NNs be optimized in an almost complete set. Our approach proves adaptable to diverse tasks, offering compatibility across various scenarios. Quantitative experimental results substantiate the potential of our structure, indicating the improvement of NNs.

Mathematical representation of the structure of neuron-glia networks

A Method for the Symbolic Representation of Neurons

The Intrinsic Similarity of Topological Structure in Biological Neural Networks.

Computational Modeling of Neuronal Networks

Multiscale modeling of brain network organization

Neural population geometry: An approach for understanding biological and artificial neural networks

Multilayer adaptive networks in neuronal processing

Information Processing by Neuron Populations in the Central Nervous System: Mathematical Structure of Data and Operations

Equivariant Matrix Function Neural Networks

Neuronal Networks in the Brain

Two's company, three (or more) is a simplex: Algebraic-topological tools for understanding higher-order structure in neural data

Interpretable Graph Neural Networks for Connectome-Based Brain Disorder Analysis

Graph Metanetworks for Processing Diverse Neural Architectures

Graph Neural Networks in Network Neuroscience

Functional Connectomics from Data: Probabilistic Graphical Models for Neuronal Network of C. elegans

Automatic Organization of Neural Modules for Enhanced Collaboration in Neural Networks

Neuroscience needs Network Science

A Graph Theoretic Interpretation of Neural Complexity

Persistence of information flow: A multiscale characterization of human brain

Non-Euclidean Spatial Graph Neural Network

Graph Element Networks: adaptive, structured computation and memory