Abstract:In the real world, multimodal sentiment analysis (MSA) enables the capture and analysis of sentiments by fusing multimodal information, thereby enhancing the understanding of real-world environments. The key challenges lie in handling the noise in the acquired data and achieving effective multimodal fusion. When processing the noise in data, existing methods utilize the combination of multimodal features to mitigate errors in sentiment word recognition caused by the performance limitations of automatic speech recognition (ASR) models. However, there still remains the problem of how to more efficiently utilize and combine different modalities to address the data noise. In multimodal fusion, most existing fusion methods have limited adaptability to the feature differences between modalities, making it difficult to capture the potential complex nonlinear interactions that may exist between modalities. To overcome the aforementioned issues, this paper proposes a new framework named multimodal-word-refinement and cross-modal-hierarchy (MWRCMH) fusion. Specifically, we utilized a multimodal word correction module to reduce sentiment word recognition errors caused by ASR. During multimodal fusion, we designed a cross-modal hierarchical fusion module that employed cross-modal attention mechanisms to fuse features between pairs of modalities, resulting in fused bimodal-feature information. Then, the obtained bimodal information and the unimodal information were fused through the nonlinear layer to obtain the final multimodal sentiment feature information. Experimental results on the MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek datasets demonstrated that the proposed approach outperformed other comparative methods, achieving Has0-F1 scores of 76.43%, 80.15%, and 81.93%, respectively. Our approach exhibited better performance, as compared to multiple baselines.

Multimodal Sensory Data Fusion-aided Channel Modelling: an End-to-End Approach

Multimodal and Multiuser Semantic Communications for Channel-Level Information Fusion

Multi-Variate Data Fusion Technique for Reducing Sensor Errors in Intelligent Transportation Systems

Deep Multimodal Learning: Merging Sensory Data for Massive MIMO Channel Prediction

Multi-Modal Intelligent Channel Modeling: A New Modeling Paradigm via Synesthesia of Machines

V2VFusion: Multimodal Fusion for Enhanced Vehicle-to-Vehicle Cooperative Perception

Study on Multi-Heterogeneous Sensor Data Fusion Method Based on Millimeter-Wave Radar and Camera

Radar and Camera Fusion for Multi-Task Sensing in Autonomous Driving

Multi-Modality Cascaded Fusion Technology for Autonomous Driving

An improved Multi-channels Information Fusion Model with multi-scale signals for fault diagnosis

Multi-Modal Fusion-Based Multi-Task Semantic Communication System

DMCE: Diffusion Model Channel Enhancer for Multi-User Semantic Communication Systems

Multimodal Sentiment Analysis in Realistic Environments Based on Cross-Modal Hierarchical Fusion Network

Multi-Modal Fusion for Enhanced Automatic Modulation Classification

Multi-modal fusion for sensing-aided beam tracking in mmWave communications

Streamlining Multimodal Data Fusion in Wireless Communication and Sensor Networks

Integrated Sensing and Communications Towards Proactive Beamforming in mmWave V2I via Multi-Modal Feature Fusion (MMFF)

Multimodal Fusion Assisted mmWave Beam Training in Dual-Model Networks

FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication

Collaborative Perception Method Based on Multisensor Fusion

A new intelligent multi-sensor data fusion framework in AFS