RoBERTa-BiLSTM: A Context-Aware Hybrid Model for Sentiment Analysis

Md. Mostafizer Rahman,Ariful Islam Shiplu,Yutaka Watanobe,Md. Ashad Alam

2024-06-01

Abstract:Effectively analyzing the comments to uncover latent intentions holds immense value in making strategic decisions across various domains. However, several challenges hinder the process of sentiment analysis including the lexical diversity exhibited in comments, the presence of long dependencies within the text, encountering unknown symbols and words, and dealing with imbalanced datasets. Moreover, existing sentiment analysis tasks mostly leveraged sequential models to encode the long dependent texts and it requires longer execution time as it processes the text sequentially. In contrast, the Transformer requires less execution time due to its parallel processing nature. In this work, we introduce a novel hybrid deep learning model, RoBERTa-BiLSTM, which combines the Robustly Optimized BERT Pretraining Approach (RoBERTa) with Bidirectional Long Short-Term Memory (BiLSTM) networks. RoBERTa is utilized to generate meaningful word embedding vectors, while BiLSTM effectively captures the contextual semantics of long-dependent texts. The RoBERTa-BiLSTM hybrid model leverages the strengths of both sequential and Transformer models to enhance performance in sentiment analysis. We conducted experiments using datasets from IMDb, Twitter US Airline, and Sentiment140 to evaluate the proposed model against existing state-of-the-art methods. Our experimental findings demonstrate that the RoBERTa-BiLSTM model surpasses baseline models (e.g., BERT, RoBERTa-base, RoBERTa-GRU, and RoBERTa-LSTM), achieving accuracies of 80.74%, 92.36%, and 82.25% on the Twitter US Airline, IMDb, and Sentiment140 datasets, respectively. Additionally, the model achieves F1-scores of 80.73%, 92.35%, and 82.25% on the same datasets, respectively.

Computation and Language,Artificial Intelligence,Computational Engineering, Finance, and Science

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to propose a new hybrid deep learning model—RoBERTa-BiLSTM for sentiment analysis tasks. Specifically:

1. **Combining the Advantages of Two Models**:
- The model integrates the strengths of the Transformer architecture (RoBERTa) and recurrent neural networks (BiLSTM) to improve sentiment analysis performance.
- RoBERTa is used to generate meaningful word embedding vectors, while BiLSTM effectively captures contextual semantics in long-dependency texts.

2. **Overcoming the Limitations of Existing Methods**:
- Existing sentiment analysis tasks mainly use sequential models to encode long-dependency texts, but these models are relatively slow in processing.
- Although Transformer models are fast in processing, they may not perform as well as sequential models in certain tasks.
- The RoBERTa-BiLSTM model combines the advantages of both models, improving both processing speed and accuracy.

3. **Improving Performance on Different Datasets**:
- The model has been experimentally validated on datasets such as IMDb, Twitter US Airline, and Sentiment140, achieving accuracies of 80.74%, 92.36%, and 82.25% respectively.
- These results indicate that the RoBERTa-BiLSTM model outperforms other baseline models (such as BERT, RoBERTa-base, RoBERTa-GRU, and RoBERTa-LSTM) across various datasets.

4. **Optimizing Hyperparameters and Handling Imbalanced Datasets**:
- To further enhance model performance, researchers tuned the model's hyperparameters and applied data augmentation to imbalanced datasets, demonstrating the performance differences before and after augmentation.

Through the above methods, the paper aims to provide an efficient and accurate sentiment analysis solution, particularly excelling in handling long-dependency texts and complex language structures.