Abstract:Active speaker detection (ASD) refers to detecting the speaking person among visible human instances in a video. Existing methods widely employed a similar audiovisual fusion approach, the concatenation. Although such a fusion approach is often argued to help enhance performance, it must be noted that neither feature modalities play an equal role. It forces the backend network to focus on learning intramodal rather than intermodal features. Another concern is that since the concatenation doubles the fused feature dimension that feeds from the audio and video module, it creates a higher computational overhead for the backend network. To address these problems, this work hypothesizes that instead of leveraging deterministic fusion operation, employing an efficient fusion technique may assist the network in learning efficiently and improve detection accuracy. This work proposes an efficient audiovisual fusion (AVF) with fewer feature dimensions that captures the correlations between facial regions and sound signals, focusing more on the discriminative facial features and associating them with the corresponding audio features. Furthermore, previous ASD works focus only on improving ASD performance by creating a large computational overhead using complex techniques such as adding sophisticated postprocessing, applying smoothing techniques on the classifier to refine the network outputs at multiple stages, or assembling the multiple network outputs. This work proposed a simple yet effective end-to-end ASD using the newly proposed feature fusion approach, the AVF. The proposed framework attained a mAP of 84.384% on the validation set of the most challenging audiovisual speaker detection benchmark, the AVA-ActiveSpeaker. With this, this work outperformed previous works that did not apply the postprocessing tasks and attained competitive detection accuracy compared to other works that employed different postprocessing tasks. The proposed model also learns better on the unsynchronized raw AVA-ActiveSpeaker dataset. The ablation experiments under different image scale settings and noisy signals show the AFV’s effectiveness and robustness than the concatenation operation.

A Speaker Tracking Algorithm Based on Audio and Visual Information Fusion Using Particle Filter

Real-Time Speaker Tracking Using Particle Filter Sensor Fusion

Object Tracking Model Based on Fusing Multiple Weighted Distribution Field Features

Efficient Audiovisual Fusion for Active Speaker Detection.

The improved particle filter for object tracking

Tracking an Object over 200 FPS with the Fusion of Prior Probability and Kalman Filter

Human Tracking Using Improved Sample-Based Joint Probabilistic Data Association Filter.

3D Audio-Visual Speaker Tracking with A Two-Layer Particle Filter.

Online Multi-Object Tracking from A Bird's-Eye View by Fusion of Millimeter-Wave Radar and Vision

An Auditory and Olfactory Data Fusion Algorithm Based on Spiking Neural Network for Mobile Robot

Visual Object Tracking Based on Motion-Adaptive Particle Filter under Complex Dynamics

Audio-Visual Speaker Tracking: Progress, Challenges, and Future Directions

Audio-Visual Bimodal Combination-Based Speaker Tracking Method for Mobile Robot

Fusion Tracking Algorithm of Mean-shift and Particle Filter Based on EMD

Visual Tracking Via Adaptive Interactive Fusion

Particle Filter Vehicles Tracking by Fusing Multiple Features.

Deep Metric Learning-Assisted 3D Audio-Visual Speaker Tracking Via Two-Layer Particle Filter

A new kind of particle filter algorithm based on immune particle swarm optimization in objective tracking

STNet: Deep Audio-Visual Fusion Network for Robust Speaker Tracking

An Improved Unscented Particle Filter Approach for Multi-Sensor Fusion Target Tracking

Target Tracking Based On A Distributed Neural Network Filtering Fusion