Abstract:Graph-based neural networks and unsupervised pre-trained models are both cutting-edge text representation methods, given their outstanding ability to capture global information and contextualized information, respectively. However, both representation methods meet obstacles to further performance improvements. On one hand, graph-based neural networks lack knowledge orientation to guide textual interpretation during global information interaction. On the other hand, unsupervised pre-trained models imply rich semantic and syntactic knowledge which lacks sufficient induction and expression. Therefore, how to effectively integrate graph-based global information and unsupervised contextualized semantic and syntactic information to achieve better text representation is an important issue pending for solution. In this paper, we propose a representation method that deeply integrates Unsupervised Semantics and Syntax into heterogeneous Graphs (USS-Graph) for inductive text classification. By constructing a heterogeneous graph whose edges and nodes are totally generated by knowledge from unsupervised pre-trained models, USS-Graph can harmonize the two perspectives of information under a bidirectionally weighted graph structure and thereby realizing the intra-fusion of graph-based global information and unsupervised contextualized semantic and syntactic information. Based on USS-Graph, we also propose a series of optimization measures to further improve the knowledge integration and representation performance. Extensive experiments conducted on benchmark datasets show that USS-Graph consistently achieves state-of-the-art performances on inductive text classification tasks. Additionally, extended experiments are conducted to deeply analyze the characteristics of USS-Graph and the effectiveness of our proposed optimization measures for further knowledge integration and information complementation.

Heterogeneous Graph Representation for Text Mining

A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sources

Heterogeneous Graph Transformer for Meta-structure Learning with Application in Text Classification

Heterogeneous graph neural networks analysis: a survey of techniques, evaluations and applications

Self-supervised Short Text Classification with Heterogeneous Graph Neural Networks

Unsupervised Extractive Summarization with Heterogeneous Graph Embeddings for Chinese Document

Learning Representation Of Heterogeneous Temporal Graphs For Recommendation

When Heterophily Meets Heterogeneous Graphs: Latent Graphs Guided Unsupervised Representation Learning

Heterogeneous Graph Generation: A Hierarchical Approach using Node Feature Pooling

Deeply Integrating Unsupervised Semantics and Syntax into Heterogeneous Graphs for Inductive Text Classification

Heterogeneous Graph Sparsification for Efficient Representation Learning

Heterogeneous Deep Graph Infomax

User-as-Graph: User Modeling with Heterogeneous Graph Pooling for News Recommendation

Hierarchical Heterogeneous Graph Representation Learning for Short Text Classification.

Learning Heterogeneous Coupling Relationships Between Non-IID Terms

Learning to Represent Programs with Heterogeneous Graphs

A Multi-Granularity Heterogeneous Graph for Extractive Text Summarization

Full-text Based Context-Rich Heterogeneous Network Mining Approach for Citation Recommendation

Heterogeneous hypergraph embedding for document recommendation

A Heterogeneous Graph to Abstract Syntax Tree Framework for Text-to-SQL

Latent neighborhood-based heterogeneous graph representation