Abstract:Text classification is an important and classic application in natural language processing (NLP). Recent studies have shown that graph neural networks (GNNs) are effective in tasks with rich structural relationships and serve as effective transductive learning approaches. Text representation learning methods based on large-scale pretraining can learn implicit but rich semantic information from text. However, few studies have comprehensively utilized the contextual semantic and structural information for Chinese text classification. Moreover, the existing GNN methods for text classification did not consider the applicability of their graph construction methods to long or short texts. In this work, we propose Chinese-BERTology-wwm-GCN, a framework that combines Chinese bidirectional encoder representations from transformers (BERT) series models with whole word masking (Chinese-BERTology-wwm) and the graph convolutional network (GCN) for Chinese text classification. When building text graph, we use documents and words as nodes to construct a heterogeneous graph for the entire corpus. Specifically, we use the term frequency-inverse document frequency (TF-IDF) to construct the word-document edge weights. For long text corpora, we propose an improved pointwise mutual information (PMI*) measure for words according to their word co-occurrence distances to represent the weights of word-word edges. For short text corpora, the co-occurrence information between words is often limited. Therefore, we utilize cosine similarity to represent the word-word edge weights. During the training stage, we effectively combine the cross-entropy and hinge losses and use them to jointly train Chinese-BERTology-wwm and GCN. Experiments show that our proposed framework significantly outperforms the baselines on three Chinese benchmark datasets and achieves good performance even with few labeled training sets.

EnvText: A Chinese text mining tool for environmental domain with advanced BERT model

Extraction of temporal information from social media messages using the BERT model

RoBERTa-wwm-ext Fine-Tuning for Chinese Text Classification

SSuieBERT: Domain Adaptation Model for Chinese Space Science Text Mining and Information Extraction

A Chinese Text Classification Method Based on BERT and Convolutional Neural Network

Feature-Enhanced Nonequilibrium Bidirectional Long Short-Term Memory Model for Chinese Text Classification

Research on Text Classification Based on BERT-BiGRU Model

BatteryDataExtractor: battery-aware text-mining software embedded with BERT models

FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining

Conceptualized Representation Learning for Chinese Biomedical Text Mining

Chinese mineral named entity recognition based on BERT model

DeText: A Deep Text Ranking Framework with BERT

Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis

Chinese text classification by combining Chinese-BERTology-wwm and GCN

ESGBERT: Language Model to Help with Classification Tasks Related to Companies Environmental, Social, and Governance Practices

Text Emotion Recognition Based on XLNet-BiGRU-Att

Leveraging BERT Language Models for Multi-Lingual ESG Issue Identification

Research on sentiment classification for netizens based on the BERT-BiLSTM-TextCNN model

BERT-Based Chinese Relation Extraction for Public Security