Unimodal and Multimodal Integrated Representation Learning Via Improved Information Bottleneck for Multimodal Sentiment Analysis

Tonghui Zhang,Changfei Dong,Jinsong Su,Haiying Zhang,Yuzheng Li
DOI: https://doi.org/10.1007/978-3-031-17120-8_44
2022-01-01
Abstract:Representation learning is a significant and challenging task in multimodal sentiment analysis (MSA). It aims to improve the performance of model by learning effective unimodal or multimodal representation. To obtain desired characteristics of representation, various constraints are proposed in previous works. However, these constraints are less concerned with the filtering of task-irrelevant information, which is highly correlated with robustness of representation. In this paper, we design a framework based on information bottleneck to filter noise information. By maximizing mutual information between pairwise unimodal representations and minimizing mutual information between unimodal representation and corresponding input, we can promote unimodal representation for including more task-relevant information and filtering out task-irrelevant information. Furthermore, attention bottleneck is embedded into the unimodal encoding process to realize the interaction between different modalities. Then, to improve the discrimination of multimodal representation, we introduce supervised contrastive learning as a constraint of multimodal representation. Last, we conduct extensive experiments on two public multimodal baseline datasets. The experimental results validate the effectiveness of our model.
What problem does this paper attempt to address?