Abstract:Deep clustering is a popular unsupervised technique for feature representation learning. We recently proposed the chunk-based DeepEmoCluster framework for speech emotion recognition (SER) to adopt the concept of deep clustering as a novel semi-supervised learning (SSL) framework, which achieved improved recognition performances over conventional reconstruction-based approaches. However, the vanilla DeepEmoCluster lacks critical sentence-level temporal information that is useful for SER tasks. This study builds upon the DeepEmoCluster framework, creating a powerful SSL approach that leverages temporal information within a sentence. We propose two sentence-level temporal modeling alternatives using either the temporal-net or the triplet loss function, resulting in a novel temporal-enhanced DeepEmoCluster framework to capture essential temporal information. The key contribution to achieving this goal is the proposed sentence-level uniform sampling strategy, which preserves the original temporal order of the data for the clustering process. An extra network module (e.g., gated recurrent unit) is utilized for the temporal-net option to encode temporal information across the data chunks. Alternatively, we can impose additional temporal constraints by using the triplet loss function while training the DeepEmoCluster framework, which does not increase model complexity. Our experimental results based on the MSP-Podcast corpus demonstrate that the proposed temporal-enhanced framework significantly outperforms the vanilla DeepEmoCluster framework and other existing SSL approaches in regression tasks for the emotional attributes arousal, dominance, and valence. The improvements are observed in fully-supervised learning or SSL implementations. Further analyses validate the effectiveness of the proposed temporal modeling, showing (1) high temporal consistency in the cluster assignment, and (2) well-separated emotional patterns in the generated clusters.

Speaker Recognition Based on Pre-Trained Model and Deep Clustering

Self-attention Based Speaker Recognition Using Cluster-Range Loss

Weighted Cluster-Range Loss and Criticality-Enhancement Loss for Speaker Recognition

Single-Channel Multi-Speaker Separation using Deep Clustering

Self Supervised Representation Learning with Deep Clustering for Acoustic Unit Discovery from Raw Speech

Speaker Cluster-Based Speaker Adaptive Training for Deep Neural Network Acoustic Modeling

A Speaker-Dependent Approach to Single-Channel Joint Speech Separation and Acoustic Modeling Based on Deep Neural Networks for Robust Recognition of Multi-Talker Speech

A Speaker-Dependent Deep Learning Approach to Joint Speech Separation and Acoustic Modeling for Multi-Talker Automatic Speech Recognition

Audio–Visual Deep Clustering for Speech Separation

Analysis of Deep Clustering as Preprocessing for Automatic Speech Recognition of Sparsely Overlapping Speech

A Speaker Recognition Method Based on Stable Learning.

Simultaneous Denoising and Dereverberation Using Deep Embedding Features

Deep Speaker Feature Learning for Text-independent Speaker Verification

Speaker recognition with two-step multi-modal deep cleansing

Discriminative Learning for Monaural Speech Separation Using Deep Embedding Features

Deep temporal clustering features for speech emotion recognition

State-Clustering Based Multiple Deep Neural Networks Modeling Approach for Speech Recognition

Speaker-Independent Lipreading with Limited Data.

Fusion of deep shallow features and models for speaker recognition

Discriminative Neural Clustering for Speaker Diarisation.

Speaker identification and localization using shuffled MFCC features and deep learning