Abstract:Acoustic and articulatory signals are naturally coupled and complementary. The challenge of acquiring articulatory data and the nonlinear ill-posedness of acoustic–articulatory conversions have resulted in previous studies on speech emotion recognition (SER) primarily relying on unidirectional acoustic–articulatory conversions. However, these studies have ignored the potential benefits of bi-directional acoustic–articulatory conversion. Addressing the problem of nonlinear ill-posedness and effectively extracting and utilizing these two modal features in SER remain open research questions. To bridge this gap, this study proposes a Bi-A2CEmo framework that simultaneously addresses the bi-directional acoustic–articulatory conversion for SER. This framework comprises three components: a Bi-MGAN that addresses the nonlinear ill-posedness problem, KCLNet that enhances the emotional attributes of the mapped features, and ResTCN-FDA that fully exploits the emotional attributes of the features. Another challenge is the absence of a parallel acoustic–articulatory emotion database. To overcome this issue, this study utilizes electromagnetic articulography (EMA) to create a multi-modal acoustic–articulatory emotion database for Mandarin Chinese called STEM-E2VA. A comparative analysis is then conducted between the proposed method and state-of-the-art models to evaluate the effectiveness of the framework. Bi-A2CEmo achieves an accuracy of 89.04% in SER, which is an improvement of 5.27% compared with the actual acoustic and articulatory features recorded by the EMA. The results for the STEM-E2VA dataset show that Bi-MGAN achieves a higher accuracy in mapping and inversion than conventional conversion networks. Visualization of the mapped features before and after enhancement reveals that KCLNet reduces the intra-class spacing while increasing the inter-class spacing of the features. ResTCN-FDA demonstrates high recognition accuracy on three publicly available datasets. The experimental results show that the proposed bi-directional acoustic–articulatory conversion framework can significantly improve the SER performance.

Speech Emotion Recognition with Complementary Acoustic Representations.

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Deep Spectrum Feature Representations for Speech Emotion Recognition

Speech Emotion Recognition Using a Dual-Channel Complementary Spectrogram and the CNN-SSAE Neutral Network

Speaker-Independent Speech Emotion Recognition Based On Cnn-Blstm And Multiple Svms

Emotion Recognition with Multimodal Transformer Fusion Framework Based on Acoustic and Lexical Information

Speech Emotion Recognition Based on Convolutional Neural Network with Attention-Based Bidirectional Long Short-Term Memory Network and Multi-Task Learning

Speech Emotion Recognition Via CNN-Transformer and Multidimensional Attention Mechanism

Speech Emotion Recognition Based on Syllable-Level Feature Extraction

Modulated Fusion using Transformer for Linguistic-Acoustic Emotion Recognition

Multimodal transformer augmented fusion for speech emotion recognition

An autoencoder-based feature level fusion for speech emotion recognition

Multi-type Features Separating Fusion Learning for Speech Emotion Recognition.

MelTrans: Mel-Spectrogram Relationship-Learning for Speech Emotion Recognition via Transformers

Speech emotion recognition with deep convolutional neural networks

Speech Emotion Recognition Based on Bi-Directional Acoustic-Articulatory Conversion

Attention Based Fully Convolutional Network for Speech Emotion Recognition

Speech Emotion Recognition From 3D Log-Mel Spectrograms With Deep Learning Network

Speech Emotion Recognition Using Mel-Frequency Cepstral Coefficients & Convolutional Neural Networks

A Feature Fusion Model with Data Augmentation for Speech Emotion Recognition

FC2VR2: Few Critical Cues-aware Voice Relationship Representation for Speech Emotion Recognition with Transformer