Abstract:Recently, multi-speaker multimedia speaker recognition (MMSR) has garnered significant attention. Although prior research primarily focused on the back-end score level fusion of audio and visual information, this study delves into innovative techniques for integrating audio and visual cues from the front-end representations of both speaker's voice and face. The first method introduces the use of visual information to estimate the number of speakers. This solution addresses the challenges of estimating speaker numbers in multi-speaker conversations, especially in noisy environments. Subsequently, agglomerative hierarchical clustering is employed for speaker diarization, proving beneficial for MMSR. This approach is termed video aiding audio fusion (VAAF). The second method innovates by introducing a ratio factor to create a multimedia vector (M-vector) which concatenates face embeddings with x-vector. This amalgamation encapsulates both audio and visual cues. The resulting M-vector is then leveraged for MMSR. We name this method as video interacting audio fusion (VIAF). Experimental results on the NIST SRE 2019 audio-visual corpus reveal that the VAAF-based MMSR achieves a 6.94% and 8.31% relative reduction in minDCF and actDCF, respectively, when benchmarked against zero-effort systems. Additionally, the VIAF-based MMSR realizes a 12.08% and 12.99% relative reduction in minDCF and actDCF, respectively, compared to systems that solely utilize face embeddings. Notably, when combining both methods, the minDCF and actDCF metrics are further optimized, reaching 0.098 and 0.102, respectively.

Orthogonal subspace combination based on the joint factor analysis for text-independent speaker recognition

Eigenchannel Space Combination Method of Joint Factor Analysis

Applying Emotional Factor Analysis And I-Vector To Emotional Speaker Recognition

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Multi-feature Combination for Speaker Recognition

Multi-resolution Time Frequency Feature and Complementary Combination for Short Utterance Speaker Recognition

Compensation of Intrinsic Variability with Factor Analysis Modeling for Robust Speaker Verification

Speaker identification by BYY automatic local factor analysis based three-level voting combination

Mixture Encoder for Joint Speech Separation and Recognition

Methods of Combining Multiple Classifiers with Different Features and Their Applications to Text-Independent Speaker Identification

I-matrix for text-independent speaker recognition

Joint Speaker Features Learning for Audio-visual Multichannel Speech Separation and Recognition

The importance of spatial and spectral information in multiple speaker tracking

Joint DOA Estimation and Dereverberation Based on Multi-Channel Linear Prediction Filtering and Azimuth Sparsity

System Combination for Short Utterance Speaker Recognition.

Joint Modeling of Accents and Acoustics for Multi-Accent Speech Recognition

Integration of audio-visual information for multi-speaker multimedia speaker recognition

A Semi-Supervised Complementary Joint Training Approach for Low-Resource Speech Recognition

Discriminant Local Information Distance Preserving Projection for Text-Independent Speaker Recognition

Jointly Recognizing Speech and Singing Voices Based on Multi-Task Audio Source Separation

Factor analysis based speaker verification using ASR