Abstract:Multimodal sentiment analysis is an emerging and critical area of research, particularly in the context of social media where users express their emotions through both text and images. However, existing methods that learn common semantic features between visual and textual modalities for multimodal sentiment analysis often overlook the importance of color information, which plays a crucial role in sentiment expression, according to psychology and art theory. This paper proposes a novel model, named ICCI, which aims to enhance multimodal sentiment analysis in social media by integrating color cues, in order to address the limitation in existing approaches. The proposed model leverages both the semantic information from image-text pairs and the color cues from images to improve the accuracy of sentiment analysis. The model comprises a feature extraction module that extracts semantic features from both images and text, as well as color features from images. Furthermore, the feature interaction module employs a cross-attention mechanism to enable the interaction of information between semantic features and color features. Finally, the label prediction module integrates all attention features to enhance multimodal sentiment analysis. The proposed model was evaluated through experiments on two widely used benchmark datasets, namely MVSA-Single and MVSA-Multiple datasets, demonstrating its effectiveness in outperforming existing methods. On MVSA-Single, ICCI achieved an accuracy of 79.33%. Similarly, on MVSA-Multiple, ICCI achieved an accuracy of 73.29%. The results underscore the importance of integrating color information for more accurate sentiment analysis in social media.

Image Search by modality analysis: A study of color semantics

Color Image Scale Model Training Image Search Modality Vector Prediction Model Training Image Database Three-color Combination Extraction Modality Vector Prediction Three-color Combinations Modality Vectors of Images Automatic Image Labeling Modality Word Corpus Projection onto Modality Space Query

An Image Retrieval System with Color Emotion Query

An image-based investigation on color associations among 100 Chinese emotion words

Kansei Image Retrieval Based on Color and Emotional Semantic Features

An Effective Image Retrieval Technique Based on Color Perception

Image Colorization with an Affective Word.

The Analysis and Annotation of Semantic Modality for Chinese Words

ColorNetVis: an Interactive Color Network Analysis System for Exploring the Color Composition of Traditional Chinese Painting

A Novel Semantics-Based Image Retrieval Method Using Similarity Measure of Multi-Level Semantics

Integrating color cues to improve multimodal sentiment analysis in social media

Semantic Mapping of Color Feature and Its Application in Content Based Image Retrieval

Colors $-$Messengers of Concepts: Visual Design Mining for Learning Color Semantics

Color Representation Method Based on Multidimensional Emotion-Semantic Space

Multimodal Image Retrieval Based on Annotation Keywords and Visual Content

Color2Vec: Web-based Modeling of Word-color Association with Sociocultural Contexts

Affective Image Colorization

Semantic Color Retrieval of Color Image Based on Fuzzy Clustering

Describing Colors, Textures and Shapes for Content Based Image Retrieval - A Survey

Affective image adjustment with a single word

Image Retrieval with EMD for New Perceptual Color Feature