Abstract:Recent advances in transformer-based architectures which are pre-trained in self-supervised manner have shown great promise in several machine learning tasks. In the audio domain, such architectures have also been successfully utilised in the field of speech emotion recognition (SER). However, existing works have not evaluated the influence of model size and pre-training data on downstream performance, and have shown limited attention to generalisation, robustness, fairness, and efficiency. The present contribution conducts a thorough analysis of these aspects on several pre-trained variants of wav2vec 2.0 and HuBERT that we fine-tuned on the dimensions arousal, dominance, and valence of MSP-Podcast, while additionally using IEMOCAP and MOSI to test cross-corpus generalisation. To the best of our knowledge, we obtain the top performance for valence prediction without use of explicit linguistic information, with a concordance correlation coefficient (CCC) of .638 on MSP-Podcast. Furthermore, our investigations reveal that transformer-based architectures are more robust to small perturbations compared to a CNN-based baseline and fair with respect to biological sex groups, but not towards individual speakers. Finally, we are the first to show that their extraordinary success on valence is based on implicit linguistic information learnt during fine-tuning of the transformer layers, which explains why they perform on-par with recent multimodal approaches that explicitly utilise textual information. Our findings collectively paint the following picture: transformer-based architectures constitute the new state-of-the-art in SER, but further advances are needed to mitigate remaining robustness and individual speaker issues. To make our findings reproducible, we release the best performing model to the community.

Speech Swin-Transformer: Exploring a Hierarchical Transformer with Shifted Windows for Speech Emotion Recognition

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Multi-Scale Temporal Transformer For Speech Emotion Recognition

Speech Keyword Spotting Method Based on Swin-Transformer Model

DWFormer: Dynamic Window transFormer for Speech Emotion Recognition

A Residual Multi-Scale Convolutional Transformer Network with Chunk-level Log-Mel Spectrograms for Speech Emotion Recognition

Dawn of the transformer era in speech emotion recognition: closing the valence gap

MelTrans: Mel-Spectrogram Relationship-Learning for Speech Emotion Recognition via Transformers

DST: Deformable Speech Transformer for Emotion Recognition

An enhanced speech emotion recognition using vision transformer

Speech Emotion Recognition Via CNN-Transformer and Multidimensional Attention Mechanism

Time-Frequency Transformer: A Novel Time Frequency Joint Learning Method for Speech Emotion Recognition

SpeechFormer: A Hierarchical Efficient Framework Incorporating the Characteristics of Speech

Speech Semantic Communication Based on Swin Transformer

Temporal Shift Module with Pretrained Representations for Speech Emotion Recognition

Hierarchical Transformer Network for Utterance-Level Emotion Recognition

A Hierarchical Transformer with Speaker Modeling for Emotion Recognition in Conversation

Spatio-Temporal Swin Transformer-based 4-D EEG Emotion Recognition.

A Window Attention Based Transformer for Automatic Speech Recognition

Emotion Neural Transducer for Fine-Grained Speech Emotion Recognition