Abstract:Graph-based neural networks and unsupervised pre-trained models are both cutting-edge text representation methods, given their outstanding ability to capture global information and contextualized information, respectively. However, both representation methods meet obstacles to further performance improvements. On one hand, graph-based neural networks lack knowledge orientation to guide textual interpretation during global information interaction. On the other hand, unsupervised pre-trained models imply rich semantic and syntactic knowledge which lacks sufficient induction and expression. Therefore, how to effectively integrate graph-based global information and unsupervised contextualized semantic and syntactic information to achieve better text representation is an important issue pending for solution. In this paper, we propose a representation method that deeply integrates Unsupervised Semantics and Syntax into heterogeneous Graphs (USS-Graph) for inductive text classification. By constructing a heterogeneous graph whose edges and nodes are totally generated by knowledge from unsupervised pre-trained models, USS-Graph can harmonize the two perspectives of information under a bidirectionally weighted graph structure and thereby realizing the intra-fusion of graph-based global information and unsupervised contextualized semantic and syntactic information. Based on USS-Graph, we also propose a series of optimization measures to further improve the knowledge integration and representation performance. Extensive experiments conducted on benchmark datasets show that USS-Graph consistently achieves state-of-the-art performances on inductive text classification tasks. Additionally, extended experiments are conducted to deeply analyze the characteristics of USS-Graph and the effectiveness of our proposed optimization measures for further knowledge integration and information complementation.

Unsupervised Graph-Text Mutual Conversion with a Unified Pretrained Language Model

INFINITY: A Simple Yet Effective Unsupervised Framework for Graph-Text Mutual Conversion

CycleGT: Unsupervised Graph-to-Text and Text-to-Graph Generation via Cycle Training

Deeply Integrating Unsupervised Semantics and Syntax into Heterogeneous Graphs for Inductive Text Classification

UniGraph: Learning a Unified Cross-Domain Foundation Model for Text-Attributed Graphs

JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs

A Pure Transformer Pretraining Framework on Text-attributed Graphs

Unifying Structured Data as Graph for Data-to-Text Pre-Training

GraphControl: Adding Conditional Control to Universal Graph Pre-trained Models for Graph Domain Transfer Learning

Contrastive Information Extraction with Generative Transformer

Deep Graph Translation

GraphGPT: Graph Learning with Generative Pre-trained Transformers

Unsupervisedly Learned Latent Graphs as Transferable Representations

Weakly Supervised Concept Map Generation through Task-Guided Graph Translation

GraphTranslator: Aligning Graph Model to Large Language Model for Open-ended Tasks

Graph Transformer for Graph-to-Sequence Learning

Tensor Graph Convolutional Networks for Text Classification

GraphGPT: Graph Instruction Tuning for Large Language Models

Heterogeneous Graph Transformer for Graph-to-Sequence Learning

Heterogeneous Graph Transformer for Meta-structure Learning with Application in Text Classification

UniGM: Unifying Multiple Pre-trained Graph Models Via Adaptive Knowledge Aggregation