Abstract:Textual graphs are ubiquitous in real-world applications, featuring rich text information with complex relationships, which enables advanced research across various fields. Textual graph representation learning aims to generate low-dimensional feature embeddings from textual graphs that can improve the performance of downstream tasks. A high-quality feature embedding should effectively capture both the structural and the textual information in a textual graph. However, most textual graph dataset benchmarks rely on word2vec techniques to generate feature embeddings, which inherently limits their capabilities. Recent works on textual graph representation learning can be categorized into two folds: supervised and unsupervised methods. Supervised methods finetune a language model on labeled nodes, which have limited capabilities when labeled data is scarce. Unsupervised methods, on the other hand, extract feature embeddings by developing complex training pipelines. To address these limitations, we propose a novel unified unsupervised learning autoencoder framework, named Node Level Graph AutoEncoder (NodeGAE). We employ language models as the backbone of the autoencoder, with pretraining on text reconstruction. Additionally, we add an auxiliary loss term to make the feature embeddings aware of the local graph structure. Our method maintains simplicity in the training process and demonstrates generalizability across diverse textual graphs and downstream tasks. We evaluate our method on two core graph representation learning downstream tasks: node classification and link prediction. Comprehensive experiments demonstrate that our approach substantially enhances the performance of diverse graph neural networks (GNNs) across multiple textual graph datasets.

Graph Convolutional Extreme Learning Machine Autoencoder for Graph Embedding

Graph convolutional autoencoders with co-learning of graph structure and node attributes

Stabilizing and Enhancing Link Prediction through Deepened Graph Auto-Encoders

Graph autoencoder for directed weighted network

Feature-Dependent Graph Convolutional Autoencoders with Adversarial Training Methods

Node Level Graph Autoencoder: Unified Pretraining for Textual Graph Learning

Keep It Simple: Graph Autoencoders Without Graph Convolutional Networks

Multi-View Graph Autoencoder for Unsupervised Graph Representation Learning

Encoder Embedding for General Graph and Node Classification

Learning Deep Graph Representations via Convolutional Neural Networks

Relating graph auto-encoders to linear models

DyGCN: Dynamic Graph Embedding with Graph Convolutional Network

Semi-AttentionAE: An Integrated Model for Graph Representation Learning

Xgcn: an Extreme Graph Convolutional Network for Large-scale Social Link Prediction

Learning Deep Representations for Graph Clustering

AutoGraph: Automated Graph Neural Network

DyGCN: Efficient Dynamic Graph Embedding With Graph Convolutional Network

Deep Manifold Graph Auto-Encoder for Attributed Graph Embedding

Graph Autoencoder with Preserving Node Attribute Similarity

Unsupervised Graph Embedding via Adaptive Graph Learning