Abstract:General language model BERT pre-trained on cross-domain text corpus, BookCorpus and Wikipedia, achieves excellent performance on a couple of natural language processing tasks through the way of fine-tuning in the downstream tasks. But it still lacks of task-specific knowledge and domain-related knowledge for further improving the performance of BERT model and more detailed fine-tuning strategy analyses are necessary. To address these problem, a BERT-based text classification model BERT4TC is proposed via constructing auxiliary sentence to turn the classification task into a binary sentence-pair one, aiming to address the limited training data problem and task-awareness problem. The architecture and implementation details of BERT4TC are also presented, as well as a post-training approach for addressing the domain challenge of BERT. Finally, extensive experiments are conducted on seven public widely-studied datasets for analyzing the fine-tuning strategies from the perspectives of learning rate, sequence length and hidden state vector selection. After that, BERT4TC models with different auxiliary sentences and post-training objectives are compared and analyzed in depth. The experiment results show that BERT4TC with suitable auxiliary sentence significantly outperforms both typical feature-based methods and fine-tuning methods, and achieves new state-of-the-art performance on multi-class classification datasets. For binary sentiment classification datasets, our BERT4TC post-trained with suitable domain-related corpus also achieves better results compared with original BERT model.

icsBERTs: Optimizing Pre-trained Language Models in Intelligent Customer Service

icsPLMs: Exploring Pre-trained Language Models in Intelligent Customer Service (Student Abstract)

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains.

Optimized Biomedical Question-Answering Services with LLM and Multi-BERT Integration

Performance Evaluations of Large Language Models for Customer Service

Application of Artificial Intelligence in Customer Service Field

Go Beyond Plain Fine-tuning: Improving Pretrained Models for Social Commonsense

Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots

Pre-training Tasks for User Intent Detection and Embedding Retrieval in E-commerce Search

SiBert: Enhanced Chinese Pre-trained Language Model with Sentence Insertion.

Optimizing Customer Satisfaction Through Sentiment Analysis: A BERT-Based Machine Learning Approach to Extract Insights

Research and Application of Large Model-Based Intelligent Customer Service System

A Data-centric Solution to Improve Online Performance of Customer Service Bots.

IEKM: A Model Incorporating External Keyword Matrices

E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce

Research on automatic labeling of imbalanced texts of customer complaints based on text enhancement and layer-by-layer semantic matching

CA-BERT: Leveraging Context Awareness for Enhanced Multi-Turn Chat Interaction

InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective

Improving BERT-Based Text Classification With Auxiliary Sentence and Domain Knowledge

SSuieBERT: Domain Adaptation Model for Chinese Space Science Text Mining and Information Extraction

Optimized Attention Enhanced Temporal Graph Convolutional Network Espoused Research of Intelligent Customer Service System based on Natural Language Processing Technology