Abstract:State-of-the-art performances for natural language processing tasks are achieved by supervised learning, specifically, by fine-tuning pre-trained language models such as BERT (Bidirectional Encoder Representation from Transformers). With increasingly accurate models, the size of the fine-tuned pre-training corpus is becoming larger and larger. However, very few studies have explored the selection of pre-training corpus. Therefore, this paper proposes a data enhancement-based domain pre-training method. At first, a pre-training task and a downstream fine-tuning task are jointly trained to alleviate the catastrophic forgetting problem generated by existing classical pre-training methods. Then, based on the hard-to-classify texts identified from downstream tasks’ feedback, the pre-training corpus can be reconstructed by selecting the similar texts from it. The learning of the reconstructed pre-training corpus can deepen the model’s understanding of undeterminable text expressions, thus enhancing the model’s feature extraction ability for domain texts. Without any pre-processing of the pre-training corpus, the experiments are conducted for two tasks, named entity recognition (NER) and text classification (CLS). The results show that learning the domain corpus selected by the proposed method can supplement the model’s understanding of domain-specific information and improve the performance of the basic pre-training model to achieve the best results compared with other benchmark methods.

Effects of Pre- and Post-Processing on type-based Embeddings in Lexical Semantic Change Detection

Explaining and Improving BERT Performance on Lexical Semantic Change Detection

Survey of Computational Approaches to Lexical Semantic Change

Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change

How Document Pre-processing affects Keyphrase Extraction Performance

Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change

Assessing the Role of Lexical Semantics in Cross-lingual Transfer through Controlled Manipulations

Effect of Post-processing on Contextualized Word Representations

Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languages

A Tale of Two Laws of Semantic Change: Predicting Synonym Changes with Distributional Semantic Models

Towards Semantics-Enhanced Pre-Training: Can Lexicon Definitions Help Learning Sentence Meanings?

IMS at SemEval-2020 Task 1: How low can you go? Dimensionality in Lexical Semantic Change Detection

Probing Pretrained Language Models for Lexical Semantics

Assessing In-context Learning and Fine-tuning for Topic Classification of German Web Data

Lexical semantic change for Ancient Greek and Latin

An Empirical Study on Post-processing Methods for Word Embeddings

Temporal Effects on Pre-trained Models for Language Processing Tasks

A Multidimensional Framework for Evaluating Lexical Semantic Change with Social Science Applications

A Joint Domain-Specific Pre-Training Method Based on Data Enhancement

Using Synchronic Definitions and Semantic Relations to Classify Semantic Change Types

Computational modeling of semantic change