Learning Trimodal Relation for AVQA with Missing Modality

Kyu Ri Park,Hong Joo Lee,Jung Uk Kim

2024-07-23

Abstract:Recent Audio-Visual Question Answering (AVQA) methods rely on complete visual and audio input to answer questions accurately. However, in real-world scenarios, issues such as device malfunctions and data transmission errors frequently result in missing audio or visual modality. In such cases, existing AVQA methods suffer significant performance degradation. In this paper, we propose a framework that ensures robust AVQA performance even when a modality is missing. First, we propose a Relation-aware Missing Modal (RMM) generator with Relation-aware Missing Modal Recalling (RMMR) loss to enhance the ability of the generator to recall missing modal information by understanding the relationships and context among the available modalities. Second, we design an Audio-Visual Relation-aware (AVR) diffusion model with Audio-Visual Enhancing (AVE) loss to further enhance audio-visual features by leveraging the relationships and shared cues between the audio-visual modalities. As a result, our method can provide accurate answers by effectively utilizing available information even when input modalities are missing. We believe our method holds potential applications not only in AVQA research but also in various multi-modal scenarios.

Computer Vision and Pattern Recognition,Artificial Intelligence,Multimedia

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the performance degradation of audio - visual question - answering (AVQA) systems in the case of incomplete multi - modal inputs. Specifically, existing AVQA methods rely on complete audio and visual inputs to answer questions accurately. However, in real - world scenarios, common problems such as device failures or data transmission errors can lead to the absence of the audio or visual modality, which causes a significant performance drop of existing AVQA methods when dealing with incomplete input modalities. To this end, the author proposes a new framework aimed at ensuring the robust performance of AVQA even when a certain modality is missing.

The main contributions of the paper include:
1. Proposing a new AVQA framework that can simultaneously address the problem of missing modalities.
2. Designing a relation - aware missing - modality generator (RMM), which recalls the information of the missing modality by correlating the two existing modalities and generates pseudo - features.
3. Introducing an audio - visual relation - aware diffusion model (AVR), which enhances the feature representation by effectively using the mutual cues between different modalities.

These innovations together ensure that the AVQA system can still provide accurate answers in the case of missing modalities, thereby improving the overall performance and robustness of the system.