Abstract:Graph Neural Networks (GNNs) have emerged as the most powerful weapon for various graph tasks due to the message-passing mechanism's great local information aggregation ability. However, over-smoothing has always hindered GNNs from going deeper and capturing multi-hop neighbors. Meanwhile, most methods follow a semi-supervised learning manner, the label scarcity would limit their applicability in real-world systems. Unlike GNNs, Transformers can model global information and multi-hop interactions via multi-head self-attention and a proper Transformer structure can show more immunity to over-smoothing. So, can we propose a novel framework to combine GNN and Transformer, integrating both GNN's local information aggregation and Transformer's global information modeling ability to eliminate the over-smoothing problem and achieve self-supervised graph representation? To realize this, this paper proposes a collaborative learning scheme for GNN-Transformer and constructs GTC architecture. GTC leverages the GNN and Transformer branch to encode node information from different views respectively, and establishes contrastive learning tasks based on the encoded cross-view information to realize self-supervised heterogeneous graph representation. For the Transformer branch, we propose Metapath-aware Hop2Token and CG-Hetphormer, which can cooperate with GNNs to attentively encode neighborhood information from different levels. As far as we know, this is the first attempt in the field of graph representation learning to utilize both GNNs and Transformer to collaboratively capture different view information and conduct cross-view contrastive learning. The experiments on real datasets show that GTC exhibits superior performance compared with state-of-the-art methods. Codes can be available at https://github.com/PHD-lanyu/GTC.

Enhancing Topological Dependencies in Spatio-Temporal Graphs with Cycle Message Passing Blocks

Mending of Spatio-Temporal Dependencies in Block Adjacency Matrix

Adaptive Spatial-Temporal Graph-Mixer for Human Motion Prediction

Boosting the Cycle Counting Power of Graph Neural Networks with I$^2$-GNNs

Do We Really Need Complicated Model Architectures For Temporal Networks?

Spatiotemporal Learning on Cell-embedded Graphs

Message Detouring: A Simple Yet Effective Cycle Representation for Expressive Graph Learning

Towards Dynamic Message Passing on Graphs

Spatial-temporal Graph Convolutional Networks with Diversified Transformation for Dynamic Graph Representation Learning

A Generalization of ViT/MLP-Mixer to Graphs

Improving the Expressiveness of $K$-hop Message-Passing GNNs by Injecting Contextualized Substructure Information

CIN++: Enhancing Topological Message Passing

Cycle Invariant Positional Encoding for Graph Representation Learning

Dynamic Graph Message Passing Networks for Visual Recognition

TopoTune : A Framework for Generalized Combinatorial Complex Neural Networks

TPGNN: Learning High-order Information in Dynamic Graphs via Temporal Propagation

GTC: GNN-Transformer co-contrastive learning for self-supervised heterogeneous graph representation

Learning to Model Graph Structural Information on MLPs via Graph Structure Self-Contrasting

ChebMixer: Efficient Graph Representation Learning with MLP Mixer

SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP