Abstract:This paper studies modulation spectrum features ($\Phi$) and mel-frequency cepstral coefficients ($\Psi$) in joint speaker diarization and identification (JSID). JSID is important as speaker diarization on its own to distinguish speakers is insufficient for many applications, it is often necessary to identify speakers as well. Machine learning models are set up using convolutional neural networks (CNNs) on $\Phi$ and recurrent neural networks $\unicode{x2013}$ long short-term memory (LSTMs) on $\Psi$, then concatenating into fully connected layers. Experiment 1 shows models on both $\Phi$ and $\Psi$ have better diarization error rates (DERs) than models on either alone; a CNN on $\Phi$ has DER 29.09\%, compared to 27.78\% for a LSTM on $\Psi$ and 19.44\% for a model on both. Experiment 1 also investigates aleatoric uncertainties and shows the model on both $\Phi$ and $\Psi$ has mean entropy 0.927~bits (out of 4~bits) for correct predictions compared to 1.896~bits for incorrect predictions which, along with entropy histogram shapes, shows the model helpfully indicates where it is uncertain. Experiment 2 investigates epistemic uncertainties as well as aleatoric using Monte Carlo dropout (MCD). It compares models on both $\Phi$ and $\Psi$ with models trained on x-vectors ($X$), before applying Kalman filter smoothing on epistemic uncertainties for resegmentation and model ensembles. While the two models on $X$ (DERs 10.23\% and 9.74\%) outperform those on $\Phi$ and $\Psi$ (DER 17.85\%) after their individual Kalman filter smoothing, combining them using a Kalman filter smoothing method improves the DER to 9.29\%. Aleatoric uncertainties are higher for incorrect predictions. Both Experiments show models on $\Phi$ do not distinguish overlapping speakers as well as anticipated. However, Experiment 2 shows model ensembles do better with overlapping speakers than individual models do.

On The Model Size Selection For Speaker Identification

Accelerate Training By Dwt In Speaker Identification Using Svms

Big model only for hard audios: Sample dependent Whisper model selection for efficient inferences

Speaker Identification Using a Reference Speaker Model Based a Two-Layer Structure

Speaker Identification Experiments Under Gender De-Identification

Noise Robust Speaker Identification Using Bhattacharyya Distance in Adapted Gaussian Models Space

Experimental evaluation of a new speaker identification framework using PCA.

Speaker Conditioning of Acoustic Models Using Affine Transformation for Multi-Speaker Speech Recognition

Robust Speaker Identification Using An Auditory-Based Feature

Sparse-Based Auditory Model for Robust Speaker Recognition

Robust Analysis And Weighting On Mfcc Components For Speech Recognition And Speaker Identification

Uncertainty Quantification in Machine Learning for Joint Speaker Diarization and Identification

Model Session Variability in Speaker Verification

Investigating model performance in language identification: beyond simple error statistics

Toward Generalizable Machine Learning Models in Speech, Language, and Hearing Sciences: Estimating Sample Size and Reducing Overfitting

A New Subspace Based Speaker Adaptation Method

Evaluating Speaker Identity Coding in Self-supervised Models and Humans

Factor analysis method for text-independent speaker identification.

Speaker Distance Estimation in Enclosures from Single-Channel Audio

Background Model Design for Flexible and Portable Speaker Verification Systems.

Subspace construction and selection for speaker recognition