Audio captioning is the task of generating captions that describe the content
of audio clips. In the real world, many objects produce similar sounds. It is
difficult to identify these auditory ambiguous sound events with access to
audio information only. How to accurately recognize ambiguous sounds is a major
challenge for audio captioning systems. In this work, inspired by the
audio-visual multi-modal perception of human beings, we propose visually-aware
audio captioning, which makes use of visual information to help the recognition
of ambiguous sounding objects. Specifically, we introduce an off-the-shelf
visual encoder to process the video inputs, and incorporate the extracted
visual features into an audio captioning system. Furthermore, to better exploit
complementary contexts from redundant audio-visual streams, we propose an
audio-visual attention mechanism that integrates audio and visual information
adaptively according to their confidence levels. Experimental results on
AudioCaps, the largest publicly available audio captioning dataset, show that
the proposed method achieves significant improvement over a strong baseline
audio captioning system and is on par with the state-of-the-art result.

本文提出了一种基于视觉感知的音频字幕生成方法，将来自视频的视觉信息整合到音频字幕系统中，采用音视频关注机制自适应地整合音频和视觉信息，显著提高了音频字幕系统的性能。