Abstract:Modern sequence models (e.g., Transformers, linear RNNs, etc.) emerged as dominant backbones of recent deep learning frameworks, mainly due to their efficiency, representational power, and/or ability to capture long-range dependencies. Adopting these sequence models for graph-structured data has recently gained popularity as the alternative to Message Passing Neural Networks (MPNNs). There is, however, a lack of a common foundation about what constitutes a good graph sequence model, and a mathematical description of the benefits and deficiencies in adopting different sequence models for learning on graphs. To this end, we first present Graph Sequence Model (GSM), a unifying framework for adopting sequence models for graphs, consisting of three main steps: (1) Tokenization, which translates the graph into a set of sequences; (2) Local Encoding, which encodes local neighborhoods around each node; and (3) Global Encoding, which employs a scalable sequence model to capture long-range dependencies within the sequences. This framework allows us to understand, evaluate, and compare the power of different sequence model backbones in graph tasks. Our theoretical evaluations of the representation power of Transformers and modern recurrent models through the lens of global and local graph tasks show that there are both negative and positive sides for both types of models. Building on this observation, we present GSM++, a fast hybrid model that uses the Hierarchical Affinity Clustering (HAC) algorithm to tokenize the graph into hierarchical sequences, and then employs a hybrid architecture of Transformer to encode these sequences. Our theoretical and experimental results support the design of GSM++, showing that GSM++ outperforms baselines in most benchmark evaluations.

What problem does this paper attempt to address?

The problems that this paper attempts to solve are: How to design an effective Graph Sequence Model (GSM) for graph - structured data, and understand the advantages and limitations of different types of sequence models when handling graph tasks. Specifically, the research aims to: 1. **Establish a unified framework**: Propose a unified Graph Sequence Model framework (GSM), which contains three main steps: Tokenization, Local Encoding, and Global Encoding. Through this framework, the performance of different sequence models in various scenarios can be systematically studied. 2. **Evaluate the advantages and disadvantages of different sequence models**: Through theoretical evaluation of the representational capabilities of Transformer and modern recurrent models (such as SSM) in global and local graph tasks, reveal the positive and negative characteristics of these models when handling graph tasks. For example, Transformers have limitations in certain tasks (such as counting tasks) due to their permutation equivariance, while SSM/RNN - based models perform well in specific tasks (such as color counting). 3. **Propose a hybrid model**: Based on the above observations, a fast hybrid model GSM++ is proposed. This model uses the Hierarchical Affinity Clustering (HAC) algorithm to tokenize the graph into a hierarchical sequence, and adopts a hybrid architecture of Transformer and SSM to encode these sequences. Experimental results show that GSM++ outperforms the baseline models in most benchmark tests. 4. **Explore the optimal tokenization strategy**: Study the impact of different tokenization strategies (nodes, edges, or sub - graphs) on model performance, and propose a new hierarchical tokenization method to provide more effective node ordering, thereby enhancing the representational ability and efficiency of the model. ### Key problem summary - **Lack of a unified basis**: Currently, there is a lack of a unified basic description of what a good graph sequence model is and the advantages and disadvantages of different sequence models in graph learning. - **Trade - off between computational complexity and representational ability**: Although the traditional Transformer is powerful, it has high computational complexity, while recurrent models (such as RNN/SSM) have better representational ability and higher computational efficiency in certain tasks. - **Complexity of graph structure**: Graph data has a complex topological structure and lacks a natural linear order, which makes it challenging to directly apply sequence models. By solving these problems, this research provides theoretical guidance and technical support for the design of future graph learning models.

Best of Both Worlds: Advantages of Hybrid Graph Sequence Models

Graph Mamba: Towards Learning on Graphs with State Space Models

Heterogeneous Graph Transformer for Graph-to-Sequence Learning

Graph Transformer for Graph-to-Sequence Learning

What Can We Learn from State Space Models for Machine Learning on Graphs?

Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges

Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling Fusion

GSRec: A Graph-Sequence Recommendation System Based on Reverse-Order Graph and User Embedding

Graph2Seq: Graph to Sequence Learning with Attention-based Neural Networks.

Multimodal Graph for Unaligned Multimodal Sequence Analysis via Graph Convolution and Graph Pooling

Hierarchically Gated Recurrent Neural Network for Sequence Modeling

To Understand Representation of Layer-aware Sequence Encoders as Multi-order-graph

Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spaces

Densely Connected Graph Convolutional Networks for Graph-to-Sequence Learning

Graph Neural Machine: A New Model for Learning with Tabular Data

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

A Dilated Recurrent Neural Network-Based Model for Graph Embedding

Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms

Transformers as Graph-to-Graph Models