Abstract:The paradigm of pre-training followed by fine-tuning on downstream tasks has become the mainstream method in natural language processing tasks. Although pre-trained models have the advantage of generalization, their performance may still vary significantly across different domain tasks. This is because the data distribution in different domains varies. For example, the different parts of the sentence 'He married Smt. Dipali Ghosh in 1947 and led a very happy married life' may have different impact for downstream tasks. For similarity calculations, words such as 'led' and 'life' are more important. On the other hand, for sentiment analysis, the word 'happy' is crucial. This indicates that different downstream tasks have different levels of sensitivity to sentence components. Our starting point is to scale information of the model and data according to the specifics of downstream tasks, enhancing domain information of relevant parts for these tasks and reducing irrelevant elements for different domain tasks, called SIFTER. In the experimental part, we use the SIFTER to improve SimCSE by constructing positive sample pairs based on enhancing the sentence stem and reducing the unimportant components in the sentence, and maximize the similarity between three sentences. Similarly, SIFTER can improve the gate mechanism of the LSTM model by short-circuiting the input gate of important words so that the LSTM model remembers the important parts of the sentence. Our experiments demonstrate that SIFTER outperforms the SimCSE and LSTM baselines.

SiBert: Enhanced Chinese Pre-trained Language Model with Sentence Insertion.

ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information

CSDR-BERT: a pre-trained scientific dataset match model for Chinese Scientific Dataset Retrieval

SLING: Sino Linguistic Evaluation of Large Language Models

What does Chinese BERT learn about syntactic knowledge?

Unsupervised Chinese Word Segmentation with BERT Oriented Probing and Transformation

Chinese text multi-classification based on Sentences Order Prediction improved Bert model

BERT Meets Chinese Word Segmentation

Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Tasks

Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models

SIFTER: A Task-specific Alignment Strategy for Enhancing Sentence Embeddings

Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots

AnchiBERT: A Pre-Trained Model for Ancient Chinese Language Understanding and Generation

RS-BERT: Pre-training radical enhanced sense embedding for Chinese word sense disambiguation

icsBERTs: Optimizing Pre-trained Language Models in Intelligent Customer Service

C2BERT - Cross-contrast BERT for Chinese Biomedical Sentence Representation.

SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models

Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models

AnchiBERT: A Pre-Trained Model for Ancient ChineseLanguage Understanding and Generation

StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

Answering Chinese Elementary School Social Study Multiple Choice Questions