Abstract:With the rapid development of multimedia technology, personnel verification systems have become increasingly important in the security field and identity verification. However, unimodal verification systems have performance bottlenecks in complex scenarios, thus triggering the need for multimodal feature fusion methods. The main problem with audio-visual multimodal feature fusion is how to effectively integrate information from different modalities to improve the accuracy and robustness of the system for individual identity. In this paper, we focus on how to improve multimodal person verification systems and how to combine audio and visual features. In this study, we use pretrained models to extract the embeddings from each modality and then perform fusion model experiments based on these embeddings. The baseline approach in this paper involves taking the fusion feature and passing it through a fully connected (FC) layer. Building upon this baseline, we propose three fusion models based on attentional mechanisms: attention, gated, and inter-attention. These fusion models are trained on the VoxCeleb1 development set and tested on the evaluation sets of the VoxCeleb1, NIST SRE19, and CNC-AV datasets. On the VoxCeleb1 dataset, the best system performance achieved in this study was an equal error rate (EER) of 0.23% and a detection cost function (minDCF) of 0.011. On the evaluation set of NIST SRE19, the EER was 2.60% and the minDCF was 0.283. On the evaluation set of the CNC-AV set, the EER was 11.30% and the minDCF was 0.443. These experimental results strongly demonstrate that the proposed fusion method can significantly improve the performance of multimodal character verification systems.

Audio-visual information fusion for SVM-based biometric verification

Fusion of Face and Visual Speech Information for Identity Verification

Audio-Visual Fusion Based on Interactive Attention for Person Verification

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Fusing face and fingerprint for identity authentication by SVM

Information Fusion in Face and Fingerprint Identity Verification System

A Multi-View Approach To Audio-Visual Speaker Verification

Multimodal Biometric Authentication Based on Score Level Fusion Using Support Vector Machine

Multimodal biometric scheme for human authentication technique based on voice and face recognition fusion

Human identification and verification based on multimodal fusion of face and ear

Android-based Multimodal Biometric Identification System Using Feature Level Fusion

A hybrid fusion model of iris, palm vein and finger vein for multi-biometric recognition system

Multi-Parts and Multi-Feature Fusion in Face Verification

Recognition Algorithm for Multi-Biometric Fusion Based on SVM

Multi-Modality Matters: A Performance Leap on VoxCeleb

A Multimodal Fusion Algorithm Based On Frr And Far Using Svm

Audio-Visual Speaker Verification via Joint Cross-Attention

An Speech and Face Fusion Recognition Method Based on Fuzzy Integral

Comparative Analysis of Modality Fusion Approaches for Audio-Visual Person Identification and Verification

Multi-level Fusion of Audio and Visual Features for Speaker Identification

A Decision Reliability Ratio Based Fusion Scheme for Biometric Verification