Abstract:Accurate recognition of human intent is crucial for effective human–computer speech interaction. Numerous intent understanding studies were based on speech-to-text transcription, which often overlook the influence of paralinguistic cues (such as speaker’s emotion, attitude, etc.), leading to misunderstandings, especially when identical textual content conveys multiple intents by means of different paralinguistic information. Considering that interaction intent is produced from the human brain, we propose a novel Multimodal Brain–Computer Fusion Network (MBCFNet) to discriminate the different intents carried by the identical textual information, in which the acoustic-textual representation was adapted by brain functional information through a cross-modal transformer. In the output module, a joint multi-task learning method was used to optimize both the primary task of intent recognition and the auxiliary task of emotion recognition of the speaker. To evaluate the model performance, we constructed a multimodal dataset CMSLIU that consists of acoustic, textual, and electroencephalograph (EEG) data of the same Chinese texts with varying intents. Experimental results on the self-constructed dataset indicate that the proposed model achieves state-of-the-art (SOTA) performance compared with other competing models. Ablation experiment results reveal that the model performance declines without information from EEG or audio modality, and also when the auxiliary task of emotion recognition is removed. All these results suggest the effectiveness of the proposed information fusion strategy in spoken language intent recognition and the brain–computer information fusion idea can also be extended to other similar fields.

A Study of Speech Intention Understanding Based on Multimodal Information Integration

Intention Understanding Based on Multi-Source Information Integration for Chinese Mandarin Spoken Commands.

Multi-Task Deep Learning for User Intention Understanding in Speech Interaction Systems

MBCFNet: A Multimodal Brain–Computer Fusion Network for Human Intention Recognition

A Multimodal Sentiment Analysis Approach Based on a Joint Chained Interactive Attention Mechanism

Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

Multimodal Affective Computing Based on Weighted Linear Fusion

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Integrative Sentiment Analysis: Leveraging Audio, Visual, and Textual Data

Integrating Audio, Visual, and Semantic Information for Enhanced Multimodal Speaker Diarization

Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment

Towards an intelligent framework for multimodal affective data analysis

Multimodal Information Processing For Affective Computing

Research And Application Analysis of Multimodal Emotion Recognition Methods Based on Speech, Text, And Facial Expressions

Multimodal integration-a statistical view

AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations

InterMulti:Multi-view Multimodal Interactions with Text-dominated Hierarchical High-order Fusion for Emotion Analysis

An Intent Recognition Model Supporting the Spoken Expression Mixed with Chinese and English.

Survey of Research on Multimodal Semantic Communication

Multimodal Sentiment Analysis With Image-Text Interaction Network