Abstract:Graphs are widely used to model interconnected entities and improve downstream predictions in various real-world applications. However, real-world graphs nowadays are often associated with complex attributes on multiple types of nodes and even links that are hard to model uniformly, while the widely used graph neural networks (GNNs) often require sufficient training toward specific downstream predictions to achieve strong performance. In this work, we take a fundamentally different approach than GNNs, to simultaneously achieve deep joint modeling of complex attributes and flexible structures of real-world graphs and obtain unsupervised generic graph representations that are not limited to specific downstream predictions. Our framework, built on a natural integration of language models (LMs) and random walks (RWs), is straightforward, powerful and data-efficient. Specifically, we first perform attributed RWs on the graph and design an automated program to compose roughly meaningful textual sequences directly from the attributed RWs; then we fine-tune an LM using the RW-based textual sequences and extract embedding vectors from the LM, which encapsulates both attribute semantics and graph structures. In our experiments, we evaluate the learned node embeddings towards different downstream prediction tasks on multiple real-world attributed graph datasets and observe significant improvements over a comprehensive set of state-of-the-art unsupervised node embedding methods. We believe this work opens a door for more sophisticated technical designs and empirical evaluations toward the leverage of LMs for the modeling of real-world graphs.

Comprehend DeepWalk as Matrix Factorization

Network Embedding As Matrix Factorization: Unifying DeepWalk, LINE, PTE, and Node2vec

Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings

DeepWalk: Online Learning of Social Representations

Word-Graph2vec: An efficient word embedding approach on word co-occurrence graph using random walk technique

Word Embedding Revisited: A New Representation Learning and Explicit Matrix Factorization Perspective.

Network Embedding for Graphs with Node Attributes

Network Representation Learning with Rich Text Information

Semantic-aware network embedding via optimized random walk and paragaraph2vec

WalkLM: A Uniform Language Model Fine-tuning Framework for Attributed Graph Embedding

To Know by the Company Words Keep and What Else Lies in the Vicinity

Fast Extraction of Word Embedding from Q-contexts

Convergence Guarantees for the DeepWalk Embedding on Block Models

Max-margin Deepwalk: Discriminative Learning of Network Representation

NetSMF: Large-Scale Network Embedding As Sparse Matrix Factorization

A Broader Picture of Random-walk Based Graph Embedding

BTWalk: Branching Tree Random Walk for Multi-Order Structured Network Embedding

Revisiting Random Walks for Learning on Graphs

Composing Knowledge Graph Embeddings via Word Embeddings

Msam: A Multi-Layer Bi-Lstm Based Speech To Vector Model With Residual Attention Mechanism

Learning Based Proximity Matrix Factorization for Node Embedding