Abstract:Graph Neural Networks (GNNs) have emerged as the most powerful weapon for various graph tasks due to the message-passing mechanism's great local information aggregation ability. However, over-smoothing has always hindered GNNs from going deeper and capturing multi-hop neighbors. Meanwhile, most methods follow a semi-supervised learning manner, the label scarcity would limit their applicability in real-world systems. Unlike GNNs, Transformers can model global information and multi-hop interactions via multi-head self-attention and a proper Transformer structure can show more immunity to over-smoothing. So, can we propose a novel framework to combine GNN and Transformer, integrating both GNN's local information aggregation and Transformer's global information modeling ability to eliminate the over-smoothing problem and achieve self-supervised graph representation? To realize this, this paper proposes a collaborative learning scheme for GNN-Transformer and constructs GTC architecture. GTC leverages the GNN and Transformer branch to encode node information from different views respectively, and establishes contrastive learning tasks based on the encoded cross-view information to realize self-supervised heterogeneous graph representation. For the Transformer branch, we propose Metapath-aware Hop2Token and CG-Hetphormer, which can cooperate with GNNs to attentively encode neighborhood information from different levels. As far as we know, this is the first attempt in the field of graph representation learning to utilize both GNNs and Transformer to collaboratively capture different view information and conduct cross-view contrastive learning. The experiments on real datasets show that GTC exhibits superior performance compared with state-of-the-art methods. Codes can be available at https://github.com/PHD-lanyu/GTC.

Self-Supervised Graph Representation Learning via Topology Transformations

Self-Supervised Node Representation Learning Via Node-to-Neighbourhood Alignment.

GraphTER: Unsupervised Learning of Graph Transformation Equivariant Representations via Auto-Encoding Node-Wise Transformations

Self-supervised Graph Representations with Generative Adversarial Learning

Topological enhanced graph neural networks for semi-supervised node classification

Generalization Guarantee of Training Graph Convolutional Networks with Graph Topology Sampling

Topology-guided Hypergraph Transformer Network: Unveiling Structural Insights for Improved Representation

Alleviating neighbor bias: augmenting graph self-supervise learning with structural equivalent positive samples

HINormer: Representation Learning on Heterogeneous Information Networks with Graph Transformer.

On the Topology Awareness and Generalization Performance of Graph Neural Networks

Topology-Informed Graph Transformer

Self-Attention Empowered Graph Convolutional Network for Structure Learning and Node Embedding

Learning to Learn Graph Topologies

Tensor-view Topological Graph Neural Network

Node Representation Learning in Graph via Node-to-Neighbourhood Mutual Information Maximization

Topology-Imbalance Learning for Semi-Supervised Node Classification

Graph Convolutional Network with elastic topology

Topological based classification using graph convolutional networks

GTC: GNN-Transformer co-contrastive learning for self-supervised heterogeneous graph representation

Understanding the Representation Power of Graph Neural Networks in Learning Graph Topology