Abstract:This work proposes a new perspective on the structure of Neural Networks (NNs). Traditional Neural Networks are typically tree-like structures for convenience, which can be predefined or learned by NAS methods. However, such a structure can not facilitate communications between nodes at the same level or signal transmissions to previous levels. These defects prevent effective collaboration, restricting the capabilities of neural networks. It is well-acknowledged that the biological neural system contains billions of neural units. Their connections are far more complicated than the current NN structure. To enhance the representational ability of neural networks, existing works try to increase the depth of the neural network and introduce more parameters. However, they all have limitations with constrained parameters. In this work, we introduce a synchronous graph-based structure to establish a novel way of organizing the neural units: the Neural Modules. This framework allows any nodes to communicate with each other and encourages neural units to work collectively, demonstrating a departure from the conventional constrained paradigm. Such a structure also provides more candidates for the NAS methods. Furthermore, we also propose an elegant regularization method to organize neural units into multiple independent, balanced neural modules systematically. This would be convenient for handling these neural modules in parallel. Compared to traditional NNs, our method unlocks the potential of NNs from tree-like structures to general graphs and makes NNs be optimized in an almost complete set. Our approach proves adaptable to diverse tasks, offering compatibility across various scenarios. Quantitative experimental results substantiate the potential of our structure, indicating the improvement of NNs.

Modular Representation of Layered Neural Networks

ModularBoost: an Efficient Network Inference Algorithm Based on Module Decomposition.

Functional Network: A Novel Framework for Interpretability of Deep Neural Networks

Graph Modularity: Towards Understanding the Cross-Layer Transition of Feature Representations in Deep Neural Networks.

Modularity in Complex Multilayer Networks with Multiple Aspects: a Static Perspective

Independent Modular Networks

Neural Network Module Decomposition and Recomposition

Knowledge Discovery from Layered Neural Networks based on Non-negative Task Decomposition

Understanding the Dynamics of DNNs Using Graph Modularity

Modular neural network via exploring category hierarchy

Modularity in Multilayer Networks Using Redundancy-Based Resolution and Projection-Based Inter-Layer Coupling

Modularity Based Community Detection with Deep Learning

Modular Boundaries in Recurrent Neural Networks

Identifying Sub-networks in Neural Networks via Functionally Similar Representations

Revisiting Resolution and Inter-Layer Coupling Factors in Modularity for Multilayer Networks

Automatic Organization of Neural Modules for Enhanced Collaboration in Neural Networks

HM-Modularity: A Harmonic Motif Modularity Approach for Multi-Layer Network Community Detection

Training Neural Networks for Modularity aids Interpretability

Modular representations emerge in neural networks trained to perform context-dependent tasks

Modularized Morphing of Neural Networks

Representing Neural Network Layers as Linear Operations via Koopman Operator Theory