Abstract:This work proposes a new perspective on the structure of Neural Networks (NNs). Traditional Neural Networks are typically tree-like structures for convenience, which can be predefined or learned by NAS methods. However, such a structure can not facilitate communications between nodes at the same level or signal transmissions to previous levels. These defects prevent effective collaboration, restricting the capabilities of neural networks. It is well-acknowledged that the biological neural system contains billions of neural units. Their connections are far more complicated than the current NN structure. To enhance the representational ability of neural networks, existing works try to increase the depth of the neural network and introduce more parameters. However, they all have limitations with constrained parameters. In this work, we introduce a synchronous graph-based structure to establish a novel way of organizing the neural units: the Neural Modules. This framework allows any nodes to communicate with each other and encourages neural units to work collectively, demonstrating a departure from the conventional constrained paradigm. Such a structure also provides more candidates for the NAS methods. Furthermore, we also propose an elegant regularization method to organize neural units into multiple independent, balanced neural modules systematically. This would be convenient for handling these neural modules in parallel. Compared to traditional NNs, our method unlocks the potential of NNs from tree-like structures to general graphs and makes NNs be optimized in an almost complete set. Our approach proves adaptable to diverse tasks, offering compatibility across various scenarios. Quantitative experimental results substantiate the potential of our structure, indicating the improvement of NNs.

Research Development on Structure-Optimisation of Forward Neural Network

Efficient Structure Slimming for Spiking Neural Networks

Optimization Algorithm Inspired Deep Neural Network Structure Design

Optimization Algorithms in Reconstructions of Neuron Morphology: an Overview

Variable Structure and Variable Learning Rate Fourier Neural Networks Research

APPLICATION OF NEURAL NETWORK IN IMAGE PROCESSING

Biologically Inspired Structure Learning with Reverse Knowledge Distillation for Spiking Neural Networks

Advances and Challenges of Optical Neural Networks

A structure optimisation algorithm for feedforward neural network construction

Structure of Artificial Neural Networks -- Empirical Investigations

Structural Optimization of Feedforward Neural Network Based on Genetic Algorithms

Research progress in optical neural networks: theory, applications and developments

Learning Structures for Deep Neural Networks

Dense Optimizer : An Information Entropy-Guided Structural Search Method for Dense-like Neural Network Design

Automatic Organization of Neural Modules for Enhanced Collaboration in Neural Networks

Online Deep Neural Network for Optimization in Wireless Communications

ANNA: Advanced neural network algorithm for optimization of structures

Universal structural patterns in sparse recurrent neural networks

On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guarantee

Strategies for Training Optical Neural Networks

A Compressed Data Partition and Loop Scheduling Scheme for Neural Networks