Abstract:Recently, multi-speaker multimedia speaker recognition (MMSR) has garnered significant attention. Although prior research primarily focused on the back-end score level fusion of audio and visual information, this study delves into innovative techniques for integrating audio and visual cues from the front-end representations of both speaker's voice and face. The first method introduces the use of visual information to estimate the number of speakers. This solution addresses the challenges of estimating speaker numbers in multi-speaker conversations, especially in noisy environments. Subsequently, agglomerative hierarchical clustering is employed for speaker diarization, proving beneficial for MMSR. This approach is termed video aiding audio fusion (VAAF). The second method innovates by introducing a ratio factor to create a multimedia vector (M-vector) which concatenates face embeddings with x-vector. This amalgamation encapsulates both audio and visual cues. The resulting M-vector is then leveraged for MMSR. We name this method as video interacting audio fusion (VIAF). Experimental results on the NIST SRE 2019 audio-visual corpus reveal that the VAAF-based MMSR achieves a 6.94% and 8.31% relative reduction in minDCF and actDCF, respectively, when benchmarked against zero-effort systems. Additionally, the VIAF-based MMSR realizes a 12.08% and 12.99% relative reduction in minDCF and actDCF, respectively, compared to systems that solely utilize face embeddings. Notably, when combining both methods, the minDCF and actDCF metrics are further optimized, reaching 0.098 and 0.102, respectively.

A Speaker Identification System for Video Content Analysis

Audio-Visual System for Robust Speaker Recognition.

A SVM-Based Audio Event Detection System

Efficient Identification Of Speakers In News Video Based On Shot Segmentation

Speaker Identification based on LSP and Gaussian Mixture Model

SVM- Ased Audio Classification for Content- Ased Multimedia Retrieval

Combined GMM-UBM and SVM Speaker Identification System

Audio Classification And Segmentation For Sports Video Structure Extraction Using Support Vector Machine

Speaker identification based on GMM feature transformation and fuzzy least-squares SVM

Design and Implementation of a Real-Time Speaker Identification System with Improved GMM

Improved SVM approach for speaker identification

Speaker Verification Based on Factor Analysis and SVM

Research and Implementation of Content Detection System for Violent Video

Pitch Synchronous Analysis Method and Fisher Criterion Based Speaker Identification.

Speaker Identification Based on Classify Feature Sub-space Gaussian Mixture Model and Neural Net Fusion

Prosodic Features-Based Speaker Verification Using Speaker-Specific-text for Short Utterances.

Speaker verification based on prosodic features

A Real Time Speaker Recognition System Based on GMM

RESEARCH ON AUDIO-BASED DIGITAL VIDEO CONTENT SAFETY DETECTION

Integration of audio-visual information for multi-speaker multimedia speaker recognition