Abstract:In aspect-based sentiment analysis (ABSA), many neural models are equipped with an attention mechanism to quantify the contribution of each context word to sentiment prediction. However, such a mechanism suffers from one drawback: only a few frequent words with sentiment polarities are tended to be taken into consideration for final sentiment decision while abundant infrequent sentiment words are ignored by models. To deal with this issue, we propose a progressive self-supervised attention learning approach for attentional ABSA models. In this approach, we iteratively perform sentiment prediction on all training instances, and continually learn useful attention supervision information in the meantime. During training, at each iteration, context words with the highest impact on sentiment prediction, identified based on their attention weights or gradients, are extracted as words with active/misleading influence on the correct/incorrect prediction for each instance. Words extracted in this way are masked for subsequent iterations. To exploit these extracted words for refining ABSA models, we augment the conventional training objective with a regularization term that encourages ABSA models to not only take full advantage of the extracted active context words but also decrease the weights of those misleading words. We integrate the proposed approach into three state-of-the-art neural ABSA models. Experiment results and in-depth analyses show that our approach yields better attention results and significantly enhances the performance of all three models. We release the source code and trained models at <a class="link-external link-https" href="https://github.com/DeepLearnXMU/PSSAttention" rel="external noopener nofollow">this https URL</a>.

Improving BERT with Self-Supervised Attention

Fine-tune BERT with Sparse Self-Attention Mechanism.

SesameBERT: Attention for Anywhere

Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation

Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guiding

Improved Visual Fine-tuning with Natural Language Supervision

Enhanced Aspect-Based Sentiment Analysis Models with Progressive Self-supervised Attention Learning

Sensi-BERT: Towards Sensitivity Driven Fine-Tuning for Parameter-Efficient BERT

Enhancing Self-Attention with Knowledge-Assisted Attention Maps

Improving BERT-Based Text Classification With Auxiliary Sentence and Domain Knowledge

Context-Guided BERT for Targeted Aspect-Based Sentiment Analysis

Improving Semantic Segmentation via Self-Training

Alleviating Over-smoothing for Unsupervised Sentence Representation.

HuBERTopic: Enhancing Semantic Representation of HuBERT through Self-supervision Utilizing Topic Model

Perturbation-based Self-supervised Attention for Attention Bias in Text Classification

BERTer: The Efficient One

A Lexicon-Based Supervised Attention Model for Neural Sentiment Analysis.

SparseBERT: Rethinking the Importance Analysis in Self-attention

Boosting Fine-Grained Sketch-Based Image Retrieval with Self-Supervised Learning

MF-BERT: Multimodal Fusion in Pre-Trained BERT for Sentiment Analysis

Classifying Scientific Publications with BERT -- Is Self-Attention a Feature Selection Method?