Cyclic Neural Network

Liangwei Yang,Hengrui Zhang,Zihe Song,Jiawei Zhang,Weizhi Zhang,Jing Ma,Philip S. Yu

2024-01-11

Abstract:This paper answers a fundamental question in artificial neural network (ANN) design: We do not need to build ANNs layer-by-layer sequentially to guarantee the Directed Acyclic Graph (DAG) property. Drawing inspiration from biological intelligence (BI), where neurons form a complex, graph-structured network, we introduce the groundbreaking Cyclic Neural Networks (Cyclic NNs). It emulates the flexible and dynamic graph nature of biological neural systems, allowing neuron connections in any graph-like structure, including cycles. This offers greater adaptability compared to the DAG structure of current ANNs. We further develop the Graph Over Multi-layer Perceptron, which is the first detailed model based on this new design paradigm. Experimental validation of the Cyclic NN's advantages on widely tested datasets in most generalized cases, demonstrating its superiority over current BP training methods through the use of a forward-forward (FF) training algorithm. This research illustrates a totally new ANN design paradigm, which is a significant departure from current ANN designs, potentially leading to more biologically plausible AI systems.

Neural and Evolutionary Computing,Machine Learning

What problem does this paper attempt to address?

The paper primarily aims to address the following core issues: 1. **Challenging the Traditional ANN Design Paradigm**: The paper explores whether it is necessary to continue using the traditional layer-by-layer construction method for artificial neural networks (ANNs) and questions whether it is truly required to build neural networks in a sequential layer-by-layer manner. 2. **Biologically Inspired New ANN Architecture**: Inspired by the complex connectivity patterns of neurons in biological neural systems, the authors propose a novel network structure—Cyclic Neural Networks. This structure allows neurons to connect in arbitrary graph structures, including forming loops. 3. **Forward-Forward Training Method**: To overcome the issue of biological implausibility, the paper employs the Forward-Forward (FF) algorithm to train cyclic neural networks. This is a localized training method where each neuron has its own loss function for optimization, eliminating the need for global error backpropagation. 4. **Validation of the New Design Paradigm**: To validate the advantages of the proposed new design paradigm, the authors designed a specific model based on cyclic neural networks—Graph Over Multi-layer Perceptron (GOMLP). Experimental results show that this model outperforms existing models trained with backpropagation on widely used datasets. In summary, this paper aims to break through the limitations of traditional layer-by-layer neural network construction by introducing cyclic neural networks and their training methods, and to explore a neural network design approach that is closer to biological intelligence.

Cyclic Neural Network

CMNN: Coupled Modular Neural Network.

Directed Acyclic Graph Neural Networks

Web Neural Network with Complete DiGraphs

Transforming to Yoked Neural Networks to Improve ANN Structure

Topological Cycle Graph Attention Network for Brain Functional Connectivity

Topographic Property of Backpropagation Artificial Neural Network: from Human Functional Connectivity Network to Artificial Neural Network

Mapping Biological Neuron Dynamics into an Interpretable Two-layer Artificial Neural Network

Biologically Inspired Structure Learning with Reverse Knowledge Distillation for Spiking Neural Networks

CircuitNet: A Generic Neural Network to Realize Universal Circuit Motif Modeling.

Dynamic Spiking Graph Neural Networks.

Causal Structure Learning With One-Dimensional Convolutional Neural Networks

GraphNAS++: Distributed Architecture Search for Graph Neural Networks

Coexistence of Cyclic Sequential Pattern Recognition and Associative Memory in Neural Networks by Attractor Mechanisms

Exploration Of A Mechanism To Form Bionic, Self-Growing And Self-Organizing Neural Network

Graph Neural Networks Inspired by Classical Iterative Algorithms

Dendrites endow artificial neural networks with accurate, robust and parameter-efficient learning

Graph Neural Machine: A New Model for Learning with Tabular Data

Cyclical Curriculum Learning

A Logical Neural Network Structure With More Direct Mapping From Logical Relations

Lifelong Learning With Cycle Memory Networks