Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention
Xubo Liu,Qiushi Huang,Xinhao Mei,Haohe Liu,Qiuqiang Kong,Jianyuan Sun,Shengchen Li,Tom Ko,Yu Zhang,Lilian H. Tang,Mark D. Plumbley,Volkan Kılıç,Wenwu Wang
DOI: https://doi.org/10.48550/arxiv.2210.16428
2022-10-28
Audio and Speech Processing
Abstract:Audio captioning is the task of generating captions that describe the content of audio clips. In the real world, many objects produce similar sounds. It is difficult to identify these auditory ambiguous sound events with access to audio information only. How to accurately recognize ambiguous sounds is a major challenge for audio captioning systems. In this work, inspired by the audio-visual multi-modal perception of human beings, we propose visually-aware audio captioning, which makes use of visual information to help the recognition of ambiguous sounding objects. Specifically, we introduce an off-the-shelf visual encoder to process the video inputs, and incorporate the extracted visual features into an audio captioning system. Furthermore, to better exploit complementary contexts from redundant audio-visual streams, we propose an audio-visual attention mechanism that integrates audio and visual information adaptively according to their confidence levels. Experimental results on AudioCaps, the largest publicly available audio captioning dataset, show that the proposed method achieves significant improvement over a strong baseline audio captioning system and is on par with the state-of-the-art result.