Abstract:The irreducible complexity of natural phenomena has led Graph Neural Networks to be employed as a standard model to perform representation learning tasks on graph-structured data. While their capacity to capture local and global patterns is remarkable, the implications associated with long-range and higher-order dependencies pose considerable challenges to such models. This work starts with a theoretical framework to reveal the impact of network's width, depth, and graph topology on the over-squashing phenomena in message-passing neural networks. Then, the work drifts towards, higher-order interactions and multi-relational inductive biases via Topological Neural Networks. Such models propagate messages through higher-dimensional structures, providing shortcuts or additional routes for information flow. With this construction, the underlying computational graph is no longer coupled with the input graph structure, thus mitigating the aforementioned bottlenecks while accounting also for higher-order interactions. Inspired by Graph Attention Networks, two topological attention networks are proposed: Simplicial and Cell Attention Networks. The rationale behind these architecture is to leverage the extended notion of neighbourhoods provided by the arrangement of groups of nodes within a simplicial or cell complex to design anisotropic aggregations able to measure the importance of the information coming from different regions of the domain. By doing so, they capture dependencies that conventional Graph Neural Networks might miss. Finally, a multi-way communication scheme is introduced with Enhanced Cellular Isomorphism Networks, which augment topological message passing schemes to enable a direct interactions among groups of nodes arranged in ring-like structures.

Non-Negative Connectivity Causes Bow-Tie Architecture in Neural Circuits

Evolution of bow-tie architectures in biology

Analyze and Design Network Architectures by Recursion Formulas

Revisiting the evolution of bow-tie architecture in signaling networks

The Intrinsic Similarity of Topological Structure in Biological Neural Networks.

Why do networks have inhibitory/negative connections?

Memorizing morph patterns in small-world neuronal network

Biologically Inspired Structure Learning with Reverse Knowledge Distillation for Spiking Neural Networks

Connectivity structure and dynamics of nonlinear recurrent neural networks

Learning Connectivity of Neural Networks from a Topological Perspective

Topological Data Analysis of Neural Network Layer Representations

Bow-tie architecture of gene regulatory networks in species of varying complexity

Deep Connectomics Networks: Neural Network Architectures Inspired by Neuronal Networks

How connectivity structure shapes rich and lazy learning in neural circuits

Understanding the wiring evolution in differentiable neural architecture search

Fixed Point Attractor Theory Bridges Structure and Function in C. elegans Neuronal Network

Topological Neural Networks: Mitigating the Bottlenecks of Graph Neural Networks via Higher-Order Interactions

Inter-layer transition in neural architecture search

Fixed Point Attractor Theory Bridges Structure and Function in <i>C. elegans</i> Neuronal Network

Learnable Brain Connectivity Structures for Identifying Neurological Disorders

Bow-tie topological features of metabolic networks and the functional significance