Abstract:Representation learning on text-attributed graphs (TAGs) has become a critical research problem in recent years. A typical example of a TAG is a paper citation graph, where the text of each paper serves as node attributes. Initial graph neural network (GNN) pipelines handled these text attributes by transforming them into shallow or hand-crafted features, such as skip-gram or bag-of-words features. Recent efforts have focused on enhancing these pipelines with language models (LMs), which typically demand intricate designs and substantial computational resources. With the advent of powerful large language models (LLMs) such as GPT or Llama2, which demonstrate an ability to reason and to utilize general knowledge, there is a growing need for techniques which combine the textual modelling abilities of LLMs with the structural learning capabilities of GNNs. Hence, in this work, we focus on leveraging LLMs to capture textual information as features, which can be used to boost GNN performance on downstream tasks. A key innovation is our use of explanations as features: we prompt an LLM to perform zero-shot classification, request textual explanations for its decision-making process, and design an LLM-to-LM interpreter to translate these explanations into informative features for downstream GNNs. Our experiments demonstrate that our method achieves state-of-the-art results on well-established TAG datasets, including Cora, PubMed, ogbn-arxiv, as well as our newly introduced dataset, tape-arxiv23. Furthermore, our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv. Lastly, we believe the versatility of the proposed method extends beyond TAGs and holds the potential to enhance other tasks involving graph-text data. Our codes and datasets are available at: <a class="link-external link-https" href="https://github.com/XiaoxinHe/TAPE" rel="external noopener nofollow">this https URL</a>.

Graph LSTM with Context-Gated Mechanism for Spoken Language Understanding.

Learning Dialogue History for Spoken Language Understanding.

Lattice LSTM for Chinese Sentence Representation

Incorporating Complete Syntactical Knowledge for Spoken Language Understanding

Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks

A Joint Model of Intent Determination and Slot Filling for Spoken Language Understanding.

Contextual LSTM (CLSTM) models for Large scale NLP tasks

Sequential Dialogue Context Modeling for Spoken Language Understanding

Deep Neural Networks Language Model Based on CNN and LSTM Hybrid Architecture

A Self-Attentive Model with Gate Mechanism for Spoken Language Understanding

Hierarchical LSTM for Sign Language Translation.

Sign Language Translation with Hierarchical Spatio-Temporal Graph Neural Network

Sign Language Translation with Hierarchical Spatio-TemporalGraph Neural Network

A Novel Slot-Gated Model Combined with a Key Verb Context Feature for Task Request Understanding by Service Robots

LSTM Acoustic Models Learn to Align and Pronounce with Graphemes

Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning

Using Bidirectional Transformer-CRF for Spoken Language Understanding.

Investigating gated recurrent neural networks for speech synthesis

Three-Module Modeling For End-to-End Spoken Language Understanding Using Pre-trained DNN-HMM-Based Acoustic-Phonetic Model

Tackling Graphical NLP problems with Graph Recurrent Networks

Long Short-Term Memory with Dynamic Skip Connections.