A Comprehensive Review of Visual-Textual Sentiment Analysis from Social Media Networks

Israa Khalaf Salman Al-Tameemi,Mohammad-Reza Feizi-Derakhshi,Saeed Pashazadeh,Mohammad Asadpour

2023-12-06

Abstract:Social media networks have become a significant aspect of people's lives, serving as a platform for their ideas, opinions and emotions. Consequently, automated sentiment analysis (SA) is critical for recognising people's feelings in ways that other information sources cannot. The analysis of these feelings revealed various applications, including brand evaluations, YouTube film reviews and healthcare applications. As social media continues to develop, people post a massive amount of information in different forms, including text, photos, audio and video. Thus, traditional SA algorithms have become limited, as they do not consider the expressiveness of other modalities. By including such characteristics from various material sources, these multimodal data streams provide new opportunities for optimising the expected results beyond text-based SA. Our study focuses on the forefront field of multimodal SA, which examines visual and textual data posted on social media networks. Many people are more likely to utilise this information to express themselves on these platforms. To serve as a resource for academics in this rapidly growing field, we introduce a comprehensive overview of textual and visual SA, including data pre-processing, feature extraction techniques, sentiment benchmark datasets, and the efficacy of multiple classification methodologies suited to each field. We also provide a brief introduction of the most frequently utilised data fusion strategies and a summary of existing research on visual-textual SA. Finally, we highlight the most significant challenges and investigate several important sentiment applications.

Computation and Language,Artificial Intelligence,Information Retrieval

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to address the issue of Multimodal Sentiment Analysis (MSA) on social media networks. Specifically, it focuses on extracting and analyzing users' sentiments from the textual and visual information posted on social media platforms.

#### Main Research Content:
1. **Review of Multimodal Sentiment Analysis**: The paper provides a comprehensive overview of textual and visual sentiment analysis, including data preprocessing, feature extraction techniques, sentiment benchmark datasets, and the effectiveness of various classification methods.
2. **Introduction to Fusion Strategies**: It briefly introduces common multimodal data fusion strategies and summarizes visual-text sentiment analysis in existing research.
3. **Challenges and Applications**: It highlights the most significant challenges in the field of sentiment analysis and explores several important sentiment applications.

#### Research Motivation:
- The rapid development of social media has made multimedia information (such as text, images, audio, and video) an important way to convey people's thoughts.
- Single-modal sentiment analysis systems have limitations in terms of accuracy, reliability, and robustness, thus necessitating the study of multimodal approaches to improve these systems.

#### Main Contributions:
- Reviewing current work to provide researchers with a complete set of methods and technical resources for visual and textual sentiment analysis.
- Providing a mathematical framework that combines multiple inputs to enhance the overall performance of sentiment analysis.
- Describing datasets, feature extraction methods, fusion methods, classification methods, and the difficulties encountered in the process of multimodal sentiment analysis.

Through the above research, the paper aims to provide the academic community with a comprehensive reference in the field of multimodal sentiment analysis, helping researchers better understand and apply the methods and techniques in this area.