Abstract:<p>Multimodal sentiment analysis is one of the most attractive interdisciplinary research topics in artificial intelligence (AI). Different from other classification issues, multimodal sentiment analysis of human is a much finer classification problem. However, most current work accept all multimodalities as the input together and then output final results at one time after fusion and decision processes. Rare models try to divide their models into more than one fusion modules with different fusion strategies for better adaption of different tasks. Additionally, most recent multimodal sentiment analysis methods pay great focuses on binary classification, but the accuracy of multi-classification still remains difficult to improve. Inspired by the emotional processing procedure in cognitive science, both binary and multi-classification abilities are improved in our method by dividing the complicated problem into smaller issues which are easier to be handled. In this paper, we propose a Hierarchal Attention-BiLSTM(Bidirectional Long-Short Term Memory) model based on Cognitive Brain limbic system (HALCB). HALCB splits the multimodal sentiment analysis into two modules responsible for two tasks, the binary classification and the multi-classification. The former module divides the input items into two categories by recognizing their polarity and then sends them to the latter module separately. In this module, Hash algorithm is utilized to improve the retrieve accuracy and speed. Correspondingly, the latter module contains a positive sub-net dedicated for positive inputs and a negative sub-nets dedicated for negative inputs. Each of these binary module and two sub-nets in multi-classification module possesses different fusion strategy and decision layer for matching its respective function. We also add a random forest at the final link to collect outputs from all modules and fuse them at the decision-level at last. Experiments are conducted on three datasets and compare the results with baselines on both binary classification and multi-classification tasks. Our experimental results surpass the state-of-the-art multimodal sentiment analysis methods on both binary and multi-classification by a big margin.</p>

Multimodal Sentiment Analysis Based on Information Bottleneck and Attention Mechanisms

Multimodal consistency-specificity fusion based on information bottleneck for sentiment analysis

Balanced sentimental information via multimodal interaction model

Sentiment Analysis of Social Media Comments Based on Multimodal Attention Fusion Network

Context-Dependent Multimodal Sentiment Analysis Based on a Complex Attention Mechanism

Multimodal Sentiment Analysis Based on Cross-Modal Attention and Gated Cyclic Hierarchical Fusion Networks

Unimodal and Multimodal Integrated Representation Learning Via Improved Information Bottleneck for Multimodal Sentiment Analysis

Multimodal sentiment analysis based on multiple attention

Multimodal Sentiment Analysis Based on Attentional Temporal Convolutional Network and Multi-Layer Feature Fusion

Multimodal sentiment analysis based on multi-head attention mechanism

A cognitive brain model for multimodal sentiment analysis based on attention neural networks

Multi-Feature Fusion Multi-Modal Sentiment Analysis Model Based on Cross-Attention Mechanism

Multimodal Sentiment Analysis Based on a Cross-Modal Multihead Attention Mechanism

Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations

Mutual information maximization and feature space separation and bi-bimodal mo-dality fusion for multimodal sentiment analysis

Sentiment Analysis Based on Text Information Enhancement and Multimodal Feature Fusion

Exploring Multimodal Sentiment Analysis via CBAM Attention and Double-layer BiLSTM Architecture

Attention-Based Modality-Gated Networks for Image-Text Sentiment Analysis

A multimodal sentiment recognition method based on attention mechanism

A Multimodal Sentiment Analysis Method Integrating Multi-Layer Attention Interaction and Multi-Feature Enhancement

Feature Extraction Network with Attention Mechanism for Data Enhancement and Recombination Fusion for Multimodal Sentiment Analysis