Abstract:Emotion recognition models using audio input data can enable the development of interactive systems with applications in mental healthcare, marketing, gaming, and social media analysis. While the field of affective computing using audio data is rich, a major barrier to achieve consistently high-performance models is the paucity of available training labels. Self-supervised learning (SSL) is a family of methods which can learn despite a scarcity of supervised labels by predicting properties of the data itself. To understand the utility of self-supervised learning for audio-based emotion recognition, we have applied self-supervised learning pre-training to the classification of emotions from the CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU- MOSEI)'s acoustic data. Unlike prior papers that have experimented with raw acoustic data, our technique has been applied to encoded acoustic data with 74 parameters of distinctive audio features at discrete timesteps. Our model is first pre-trained to uncover the randomly masked timestamps of the acoustic data. The pre-trained model is then fine-tuned using a small sample of annotated data. The performance of the final model is then evaluated via overall mean absolute error (MAE), mean absolute error (MAE) per emotion, overall four-class accuracy, and four-class accuracy per emotion. These metrics are compared against a baseline deep learning model with an identical backbone architecture. We find that self-supervised learning consistently improves the performance of the model across all metrics, especially when the number of annotated data points in the fine-tuning step is small. Furthermore, we quantify the behaviors of the self-supervised model and its convergence as the amount of annotated data increases. This work characterizes the utility of self-supervised learning for affective computing, demonstrating that self-supervised learning is most useful when the number of training examples is small and that the effect is most pronounced for emotions which are easier to classify such as happy, sad, and angry. This work further demonstrates that self-supervised learning still improves performance when applied to the embedded feature representations rather than the traditional approach of pre-training on the raw input space.

Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognition

Evaluating Self-Supervised Speech Representations for Speech Emotion Recognition

Audio-Based Emotion Recognition Using Self-Supervised Learning on an Engineered Feature Space

Speaker-Independent Speech Emotion Recognition Based On Cnn-Blstm And Multiple Svms

ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognition

Wav2vec2. 0 and Context Emotional Information Compensation Based Dialogue Speech Emotion Recognition

emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation

Speech Emotion Recognition Using Self-Supervised Features

Adapting WavLM for Speech Emotion Recognition

Self-supervised utterance order prediction for emotion recognition in conversations

Probing self-supervised speech models for phonetic and phonemic information: a case study in aspiration

Self-Supervised Learning for Audio-Based Emotion Recognition

Unsupervised Representations Improve Supervised Learning in Speech Emotion Recognition

Learning Fine-Grained Cross Modality Excitement for Speech Emotion Recognition

Semi-Supervised Learning for Robust Emotional Speech Synthesis with Limited Data

A Comparative Study of Self-Supervised Speech Representations in Read and Spontaneous TTS

Supervised Contrastive Learning with Nearest Neighbor Search for Speech Emotion Recognition

A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding

Emotion-Aware Speech Self-Supervised Representation Learning with Intensity Knowledge