Sentiment Analysis of Lithuanian Online Reviews Using Large Language Models

Brigita Vileikytė,Mantas Lukoševičius,Lukas Stankevičius

2024-07-29

Abstract:Sentiment analysis is a widely researched area within Natural Language Processing (NLP), attracting significant interest due to the advent of automated solutions. Despite this, the task remains challenging because of the inherent complexity of languages and the subjective nature of sentiments. It is even more challenging for less-studied and less-resourced languages such as Lithuanian. Our review of existing Lithuanian NLP research reveals that traditional machine learning methods and classification algorithms have limited effectiveness for the task. In this work, we address sentiment analysis of Lithuanian five-star-based online reviews from multiple domains that we collect and clean. We apply transformer models to this task for the first time, exploring the capabilities of pre-trained multilingual Large Language Models (LLMs), specifically focusing on fine-tuning BERT and T5 models. Given the inherent difficulty of the task, the fine-tuned models perform quite well, especially when the sentiments themselves are less ambiguous: 80.74% and 89.61% testing recognition accuracy of the most popular one- and five-star reviews respectively. They significantly outperform current commercial state-of-the-art general-purpose LLM GPT-4. We openly share our fine-tuned LLMs online.

Computation and Language,Information Retrieval,Machine Learning

What problem does this paper attempt to address?

The paper aims to address the issue of sentiment analysis for Lithuanian online reviews. Specifically, the paper focuses on the following points:

1. **Challenges in Sentiment Analysis**: Although sentiment analysis research in the field of Natural Language Processing (NLP) has been widely conducted, this task remains challenging due to the complexity of language and the subjectivity of emotions. This challenge is particularly significant for low-resource languages like Lithuanian.

2. **Limitations of Existing Methods**: Traditional machine learning methods and classification algorithms have limited effectiveness in Lithuanian sentiment analysis, necessitating the exploration of new methods to improve accuracy.

3. **Application of Large Language Models**: The paper attempts for the first time to use pre-trained multilingual large language models (LLMs), specifically BERT and T5 models, for sentiment classification of Lithuanian five-star online reviews. Experimental results show that these models perform well in cases where the sentiment is relatively clear, achieving test recognition accuracies of 80.74% and 89.61%, respectively.

4. **Dataset Construction and Processing**: To simulate real-world scenarios, the paper collected Lithuanian review data on various topics from multiple external sources and conducted data cleaning and preprocessing to improve data quality.

5. **Performance Evaluation and Comparison**: The paper compared the performance of DistilBERT and ByT5 models on different datasets and found that the DistilBERT model achieved the best F1 score (67.51%) on the first dataset. Additionally, a commercial GPT-4 model was used for comparison, and the results showed that GPT-4 performed worse in Lithuanian sentiment classification compared to the fine-tuned DistilBERT and ByT5 models.

Through the above research, the paper demonstrates the effectiveness of large language models for Lithuanian sentiment analysis and provides valuable references for subsequent research.