Abstract:Sentiment classification is a form of data analytics where people's feelings and attitudes toward a topic are mined from data. This tantalizing power to "predict the zeitgeist" means that sentiment classification has long attracted interest, but with mixed results. However, the recent development of the BERT framework and its pretrained neural language models is seeing new-found success for sentiment classification. BERT models are trained to capture word-level information via mask language modeling and sentence-level contexts via next sentence prediction tasks. Out of the box, they are adequate models for some natural language processing tasks. However, most models are further fine-tuned with domain-specific information to increase accuracy and usefulness. Motivated by the idea that a further fine-tuning step would improve the performance for downstream sentiment classification tasks, we developed TopicBERT-a BERT model fine-tuned to recognize topics at the corpus level in addition to the word and sentence levels. TopicBERT comprises two variants: TopicBERT-ATP (aspect topic prediction), which captures topic information via an auxiliary training task, and TopicBERT-TA, where topic representation is directly injected into a topic augmentation layer for sentiment classification. With TopicBERT-ATP, the topics are predetermined by an LDA mechanism and collapsed Gibbs sampling. With TopicBERT-TA, the topics can change dynamically during the training. Experimental results show that both approaches deliver the state-of-the-art performance in two different domains with SemEval 2014 Task 4. However, in a test of methods, direct augmentation outperforms further training. Comprehensive analyses in the form of ablation, parameter, and complexity studies accompany the results.

Methods to Enhance BERT in Aspect-Based Sentiment Classification

Prior-Bert and Multi-Task Learning for Target-Aspect-Sentiment Joint Detection

Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference

TopicBERT: A Topic-Enhanced Neural Language Model Fine-Tuned for Sentiment Classification

Combining Fine-Tuning with a Feature-Based Approach for Aspect Extraction on Reviews (Student Abstract).

Fine-tune BERT with Sparse Self-Attention Mechanism.

Aspect-Level Sentiment Analysis Approach via BERT and Aspect Feature Location Model

How to Fine-Tune BERT for Text Classification?

BERTer: The Efficient One

Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation

Fine-Tuning BERT for Sentiment Analysis of Vietnamese Reviews

An improved aspect-category sentiment analysis model for text sentiment analysis based on RoBERTa

Fine-tuning Pretrained Multilingual BERT Model for Indonesian Aspect-based Sentiment Analysis

Improving BERT-Based Text Classification With Auxiliary Sentence and Domain Knowledge

Improving BERT Performance for Aspect-Based Sentiment Analysis

Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification

Implicit Aspect Extraction from Online Clothing Reviews with Fine-tuning BERT Algorithm

Research on the Application of Deep Learning-based BERT Model in Sentiment Analysis

MF-BERT: Multimodal Fusion in Pre-Trained BERT for Sentiment Analysis

Can Fine-tuning Pre-trained Models Lead to Perfect NLP? A Study of the Generalizability of Relation Extraction.

Improving Cross-Domain Aspect-Based Sentiment Analysis using Bert-BiLSTM Model and Dual Attention Mechanism