Abstract:Spectrogram is commonly used as the input feature of deep neural networks to learn the high(er)-level time-frequency pattern of speech signal for speech emotion recognition (SER). \textcolor{black}{Generally, different emotions correspond to specific energy activations both within frequency bands and time frames on spectrogram, which indicates the frequency and time domains are both essential to represent the emotion for SER. However, recent spectrogram-based works mainly focus on modeling the long-term dependency in time domain, leading to these methods encountering the following two issues: (1) neglecting to model the emotion-related correlations within frequency domain during the time-frequency joint learning; (2) ignoring to capture the specific frequency bands associated with emotions.} To cope with the issues, we propose an attentive time-frequency neural network (ATFNN) for SER, including a time-frequency neural network (TFNN) and time-frequency attention. Specifically, aiming at the first issue, we design a TFNN with a frequency-domain encoder (F-Encoder) based on the Transformer encoder and a time-domain encoder (T-Encoder) based on the Bidirectional Long Short-Term Memory (Bi-LSTM). The F-Encoder and T-Encoder model the correlations within frequency bands and time frames, respectively, and they are embedded into a time-frequency joint learning strategy to obtain the time-frequency patterns for speech emotions. Moreover, to handle the second issue, we also adopt time-frequency attention with a frequency-attention network (F-Attention) and a time-attention network (T-Attention) to focus on the emotion-related frequency band ranges and time frame ranges, which can enhance the discriminability of speech emotion features.

Deep Converged Network for Attention

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

Residual Fusion Network Based Attention for Aspect Level Sentiment Analysis

Convolution-Based Neural Attention with Applications to Sentiment Classification

Speech Emotion Recognition via an Attentive Time-Frequency Neural Network

Feature-enhanced Attention Network for Target-Dependent Sentiment Classification

Feature Extraction Network with Attention Mechanism for Data Enhancement and Recombination Fusion for Multimodal Sentiment Analysis

Deep Emotional Arousal Network for Multimodal Sentiment Analysis and Emotion Recognition

Extract, Attend, Predict: Aspect-Based Sentiment Analysis with Deep Self-Attention Network

A transformer-encoder-based multimodal multi-attention fusion network for sentiment analysis

FAN: Focused Attention Networks

Branch-Fusion-Net for Multi-Modal Continuous Dimensional Emotion Recognition

A Deep Neural Network Model for Target-based Sentiment Analysis.

Sentiment Lexicon Enhanced Neural Sentiment Classification

Building Extraction from Very High Resolution Aerial Imagery Using Joint Attention Deep Neural Network

FAT-DeepFFM: Field Attentive Deep Field-aware Factorization Machine

Aspect-Based Sentiment Analysis on Convolution Neural Network and Multi-Hierarchical Attention.

Text Sentiment Analysis of Fusion Model Based on Attention Mechanism.

Interactive Fusion Network with Recurrent Attention for Multimodal Aspect-based Sentiment Analysis.

Combined CNN LSTM with attention for speech emotion recognition based on feature-level fusion

Dilated Residual Network with Multi-head Self-attention for Speech Emotion Recognition