Abstract:Speech Emotion Recognition (SER) is an important but challenging research topic in Human–Computer interaction (HCI). As a crucial component influencing the model’s performance, in SER, feature extraction needs to comprehensively consider the static local states, the dynamic change tendency, and the time–frequency characteristics of speech signals to generate meaningful emotional representations and further improve the SER performance. To address this aim, we propose the Static and Dynamic Time–frequency Network called SDTF-Net. To reduce the interference between different features, in SDTF-Net, we introduce two parallel modules, i.e., Time–Frequency Convolutional Neural Network (TF-CNN) and Time–Frequency Transformer (TF-Trans), to generate the static and dynamic emotional features based on the time–frequency information, respectively. Specifically, in the TF-Trans module, we design a three-step strategy to learn the relatively complicated dynamic feature, comprising two parallel Bi-direction Multi-scale Dynamic Activation (BMDA) blocks, two parallel transformer (Trans) blocks, and a time–frequency fusion (TF-Cross) block. Subsequently, we propose an SD-Cross module to effectively fuse the learned static and dynamic features, generating the final meaningful emotional representations. Given the problem of data sparsity, moreover, we employ the data augmentation method called SpecAugment to improve the learning ability of SDTF-Net. From the experimental results on IEMOCAP and MSP-IMPROV datasets, we observe that SDTF-Net outperforms the state-of-the-art methods. Furthermore, extensive ablation experiments further indicate the rationality of the SDTF-Net structure and the necessity of considering dynamic and static features based on time–frequency information comprehensively in SER.

DSTCNet: Deep Spectro-Temporal-Channel Attention Network for Speech Emotion Recognition.

Attention-Enhanced Connectionist Temporal Classification for Discrete Speech Emotion Recognition

Self-attention Transfer Networks for Speech Emotion Recognition

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Deep Spectrum Feature Representations for Speech Emotion Recognition

Representation Learning with Spectro-Temporal-Channel Attention for Speech Emotion Recognition

CM-TCN: Channel-Aware Multi-scale Temporal Convolutional Networks for Speech Emotion Recognition.

Speech Emotion Recognition Based on Convolutional Neural Network with Attention-Based Bidirectional Long Short-Term Memory Network and Multi-Task Learning

Pre-trained Deep Convolution Neural Network Model With Attention for Speech Emotion Recognition

Combining a parallel 2D CNN with a self-attention Dilated Residual Network for CTC-based discrete speech emotion recognition

Speech Emotion Recognition Via CNN-Transformer and Multidimensional Attention Mechanism

Temporal Attention Convolutional Network for Speech Emotion Recognition with Latent Representation.

SDTF-Net: Static and Dynamic Time–frequency Network for Speech Emotion Recognition

ATDA: Attentional temporal dynamic activation for speech emotion recognition

Attention Based Fully Convolutional Network for Speech Emotion Recognition

3-D Convolutional Recurrent Neural Networks with Attention Model for Speech Emotion Recognition.

Speech Emotion Recognition Using Deep Convolutional Neural Network and Discriminant Temporal Pyramid Matching

STSNet: a Novel Spatio-Temporal-spectral Network for Subject-Independent EEG-based Emotion Recognition

DST: Deformable Speech Transformer for Emotion Recognition

GM-TCNet: Gated Multi-scale Temporal Convolutional Network using Emotion Causality for Speech Emotion Recognition

Speech Emotion Recognition via an Attentive Time-Frequency Neural Network