Abstract:Recent advances in transformer-based architectures which are pre-trained in self-supervised manner have shown great promise in several machine learning tasks. In the audio domain, such architectures have also been successfully utilised in the field of speech emotion recognition (SER). However, existing works have not evaluated the influence of model size and pre-training data on downstream performance, and have shown limited attention to generalisation, robustness, fairness, and efficiency. The present contribution conducts a thorough analysis of these aspects on several pre-trained variants of wav2vec 2.0 and HuBERT that we fine-tuned on the dimensions arousal, dominance, and valence of MSP-Podcast, while additionally using IEMOCAP and MOSI to test cross-corpus generalisation. To the best of our knowledge, we obtain the top performance for valence prediction without use of explicit linguistic information, with a concordance correlation coefficient (CCC) of .638 on MSP-Podcast. Furthermore, our investigations reveal that transformer-based architectures are more robust to small perturbations compared to a CNN-based baseline and fair with respect to biological sex groups, but not towards individual speakers. Finally, we are the first to show that their extraordinary success on valence is based on implicit linguistic information learnt during fine-tuning of the transformer layers, which explains why they perform on-par with recent multimodal approaches that explicitly utilise textual information. Our findings collectively paint the following picture: transformer-based architectures constitute the new state-of-the-art in SER, but further advances are needed to mitigate remaining robustness and individual speaker issues. To make our findings reproducible, we release the best performing model to the community.

Integrating Contrastive Learning into a Multitask Transformer Model for Effective Domain Adaptation

Adversarial Domain Generalized Transformer for Cross-Corpus Speech Emotion Recognition

Speech Emotion Recognition Based on Convolutional Neural Network with Attention-Based Bidirectional Long Short-Term Memory Network and Multi-Task Learning

Dynamic Transformer Architecture for Continual Learning of Multimodal Tasks

Multi-Task Transformer with Adaptive Cross-Entropy Loss for Multi-Dialect Speech Recognition

Combining Denoising Autoencoders with Contrastive Learning to fine-tune Transformer Models

Transferability of Representations Learned using Supervised Contrastive Learning Trained on a Multi-Domain Dataset

Tri-CLT: Learning Tri-Modal Representations with Contrastive Learning and Transformer for Multimodal Sentiment Recognition

Multilevel Transformer For Multimodal Emotion Recognition

Extreme Multi-Domain, Multi-Task Learning With Unified Text-to-Text Transfer Transformers

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Multitask Learning and Joint Optimization for Transformer-RNN-Transducer Speech Recognition

Task-Attentive Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillation

Bi-Modal Bi-Task Emotion Recognition Based on Transformer Architecture

Contrastive Adversarial Training for Unsupervised Domain Adaptation

Multi-Scale Temporal Transformer For Speech Emotion Recognition

TADA: Efficient Task-Agnostic Domain Adaptation for Transformers

Improving Transformer Based End-to-End Code-Switching Speech Recognition Using Language Identification

Enhancing cross-domain detection: adaptive class-aware contrastive transformer

Multimodal Transformer with Learnable Frontend and Self Attention for Emotion Recognition

Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data