Abstract:Topological Deep Learning seeks to enhance the predictive performance of neural network models by harnessing topological structures in input data. Topological neural networks operate on spaces such as cell complexes and hypergraphs, that can be seen as generalizations of graphs. In this work, we introduce the Cellular Transformer (CT), a novel architecture that generalizes graph-based transformers to cell complexes. First, we propose a new formulation of the usual self- and cross-attention mechanisms, tailored to leverage incidence relations in cell complexes, e.g., edge-face and node-edge relations. Additionally, we propose a set of topological positional encodings specifically designed for cell complexes. By transforming three graph datasets into cell complex datasets, our experiments reveal that CT not only achieves state-of-the-art performance, but it does so without the need for more complex enhancements such as virtual nodes, in-domain structural encodings, or graph rewiring.

What problem does this paper attempt to address?

This paper proposes a new architecture called Cellular Transformer (CT), which extends the Transformer model to cellular complexes to handle data with topological structures. Cellular complexes are a generalization of graphs that can represent higher-order relationships beyond the binary relationship limitation of traditional graph neural networks. Based on existing work, the paper introduces new forms of self-attention and cross-attention specifically tailored for cellular complexes, and designs specific topological positional encoding. The main contributions of the paper are as follows: 1. Proposing the CT framework, which extends the graph-based Transformer to handle higher-order relationships in cellular complexes. 2. Defining positional encoding for cellular complexes and re-describing self-attention and cross-attention in topological terms, demonstrating how to process data on cellular complexes in Transformer layers. 3. Benchmarking CT on three graph datasets, the results show that CT achieves results matching or surpassing state-of-the-art models without the need for additional complex enhancements such as virtual nodes, intra-domain structure encoding, or graph rewiring. The paper also compares existing graph transformers, higher-order transformers, and other topological neural network methods, highlighting the advantages of CT in capturing complex patterns in the data. In addition, it explores different types of attention mechanisms (dual-cell attention and general cell attention) and different positional encoding techniques, such as Baricentric subdivision encoding, random walk encoding, and topological skelepin encoding. In the experimental section, the paper uses the ZINC, ogbg-molhiv, and Graph Classification Benchmark (GCB) datasets, converts graphs into cellular complexes, and demonstrates the performance of CT on these datasets. Compared to other graph and simple complex transformers, CT reduces reliance on complex enhancements while maintaining or improving performance.

Attending to Topological Spaces: The Cellular Transformer

Topology-guided Hypergraph Transformer Network: Unveiling Structural Insights for Improved Representation

Learning Connectivity of Neural Networks from a Topological Perspective

Experimental Observations of the Topology of Convolutional Neural Network Activations

Topological Neural Networks: Mitigating the Bottlenecks of Graph Neural Networks via Higher-Order Interactions

Topological Blind Spots: Understanding and Extending Topological Deep Learning Through the Lens of Expressivity

NeuroPath: A Neural Pathway Transformer for Joining the Dots of Human Connectomes

The Topos of Transformer Networks

Reach the Remote Neighbors: Dual-Encoding Transformer for Graphs

Topology-Guided Multi-Class Cell Context Generation for Digital Pathology

Topgformer: Topological-Based Graph Transformer for Mapping Brain Structural Connectivity to Functional Connectivity

TopoTune : A Framework for Generalized Combinatorial Complex Neural Networks

EGCT: Enhanced Graph Convolutional Transformer for 3D Point Cloud Representation Learning

Topological deep learning: a review of an emerging paradigm

Topology-Informed Graph Transformer

E(n) Equivariant Topological Neural Networks

TopologyNet: Topology based deep convolutional neural networks for biomolecular property predictions

The Topological BERT: Transforming Attention into Topology for Natural Language Processing

Architectures of Topological Deep Learning: A Survey of Message-Passing Topological Neural Networks

MTS-Net: an Enriched Topology-Aware Architecture for Molecular Graph Representation Learning