Abstract:Emotion recognition models using audio input data can enable the development of interactive systems with applications in mental healthcare, marketing, gaming, and social media analysis. While the field of affective computing using audio data is rich, a major barrier to achieve consistently high-performance models is the paucity of available training labels. Self-supervised learning (SSL) is a family of methods which can learn despite a scarcity of supervised labels by predicting properties of the data itself. To understand the utility of self-supervised learning for audio-based emotion recognition, we have applied self-supervised learning pre-training to the classification of emotions from the CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU- MOSEI)'s acoustic data. Unlike prior papers that have experimented with raw acoustic data, our technique has been applied to encoded acoustic data with 74 parameters of distinctive audio features at discrete timesteps. Our model is first pre-trained to uncover the randomly masked timestamps of the acoustic data. The pre-trained model is then fine-tuned using a small sample of annotated data. The performance of the final model is then evaluated via overall mean absolute error (MAE), mean absolute error (MAE) per emotion, overall four-class accuracy, and four-class accuracy per emotion. These metrics are compared against a baseline deep learning model with an identical backbone architecture. We find that self-supervised learning consistently improves the performance of the model across all metrics, especially when the number of annotated data points in the fine-tuning step is small. Furthermore, we quantify the behaviors of the self-supervised model and its convergence as the amount of annotated data increases. This work characterizes the utility of self-supervised learning for affective computing, demonstrating that self-supervised learning is most useful when the number of training examples is small and that the effect is most pronounced for emotions which are easier to classify such as happy, sad, and angry. This work further demonstrates that self-supervised learning still improves performance when applied to the embedded feature representations rather than the traditional approach of pre-training on the raw input space.

Machine Learning Framework for Audio-Based Content Evaluation using MFCC, Chroma, Spectral Contrast, and Temporal Feature Engineering

Audio Sentiment Analysis by Heterogeneous Signal Features Learned from Utterance-Based Parallel Neural Network.

Utterance-Based Audio Sentiment Analysis Learned by a Parallel Combination of CNN and LSTM.

A multimodal deep learning algorithm for polyphonic music applied to music sentiment analysis and generation

A Comparative Study of Teaching Effectiveness in Emotionally Empowered Music Classrooms from a Multimodal Perspective

Exploring and Applying Audio-Based Sentiment Analysis in Music

Integrative Sentiment Analysis: Leveraging Audio, Visual, and Textual Data

Enhancing Music Mood Recognition with LLMs and Audio Signal Processing: A Multimodal Approach

Learning Robust Heterogeneous Signal Features from Parallel Neural Network for Audio Sentiment Analysis

Joint sentiment analysis of lyrics and audio in music

Real-Time Human-Music Emotional Interaction Based on Deep Learning and Multimodal Sentiment Analysis

Evaluating Sentiment Similarity Of Songs Based On Social Media Data

Audio-Based Emotion Recognition Using Self-Supervised Learning on an Engineered Feature Space

A Music Classification Model based on Metric Learning and Feature Extraction from MP3 Audio Files

Music sentiment classification integrating audio with lyrics

Fine-grained Sentiment Semantic Analysis and Matching of Music and Image

Real-time Human-Music Emotional Interaction Based on Multimodal Analysis

Sentiment analysis in non-fixed length audios using a Fully Convolutional Neural Network

Learning to Generate Music With Sentiment

An artificial intelligence-based classifier for musical emotion expression in media education

A Comparison Study of Deep Learning Methodologies for Music Emotion Recognition