Abstract:Human perception systems can integrate audio and visual information automatically to obtain a profound understanding of real-world events. Accordingly, fusing audio and visual contents is important to solve the audio-visual event (AVE) localization problem. Although most existing works have fused audio and visual modalities to explore their relationship with attention-based networks, we can delve into their relationship more deeply to improve the fusion capability of the two modalities. In this paper, we propose a dense modality interaction network (DMIN) to elegantly leverage audio and visual information by integrating two novel modules, namely, the audio-guided triplet attention (AGTA) module and the dense inter-modality attention (DIMA) module. The AGTA module enables audio information to guide the network to pay more attention to event-relevant visual regions. This guidance is conducted in the channel, temporal, and spatial dimensions, which emphasize informative features, temporal relationships and spatial regions, to boost the capacity of representations. Furthermore, the DIMA module establishes the dense-relationship between audio and visual modalities. Specifically, the DIMA module leverages the information of all channel pairs of audio and visual features to formulate the cross-modality attention weight, which is superior to the multi-head attention module that uses limited information. Moreover, a novel unimodal discrimination loss (UDL) is introduced to exploit the unimodal and fused features together for more exact AVE localization. The experimental results show that our method is remarkably superior to the state-of-the-art methods in fully- and weakly-supervised AVE settings. To further evaluate the models ability to build audio-visual connections, we design a dense cross modality relation network (DCMR) to solve the cross-modality localization task. DCMR is a simple deformation of a DMIN, and the experimental results further illustrate that DIMA can explore denser relationships between the two modalities. Code is available at https://github.com/weizequan/DMIN.git.

Learning Event-Specific Localization Preferences for Audio-Visual Event Localization

Specialty may be better: A decoupling multi-modal fusion network for Audio-visual event localization

Audio-Visual Event Localization in Unconstrained Videos

Masked Co-Attention Model for Audio-Visual Event Localization

Audio-Visual Event Localization by Learning Spatial and Semantic Co-attention

Past and Future Motion Guided Network for Audio Visual Event Localization

CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization

Temporal Label-Refinement for Weakly-Supervised Audio-Visual Event Localization

UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization

Leveraging the Video-level Semantic Consistency of Event for Audio-visual Event Localization

Audio-Visual Event Localization using Multi-task Hybrid Attention Networks for Smart Healthcare Systems

Visual Localizer: Outdoor Localization Based on ConvNet Descriptor and Global Optimization for Visually Impaired Pedestrians

Audio-Visual Event Localization Via Recursive Fusion by Joint Co-Attention

Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline

Dual Attention Matching for Audio-Visual Event Localization.

Fusion of Audio and Visual Embeddings for Sound Event Localization and Detection

CSS-Net: A Consistent Segment Selection Network for Audio-visual Event Localization

MPN: Multimodal Parallel Network for Audio-Visual Event Localization

AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization

Dense Modality Interaction Network for Audio-Visual Event Localization

A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition