This paper focuses on the Audio-Visual Question Answering (AVQA) task that
aims to answer questions derived from untrimmed audible videos. To generate
accurate answers, an AVQA model is expected to find the most informative
audio-visual clues relevant to the given questions. In this paper, we propose
to explicitly consider fine-grained visual objects in video frames
(object-level clues) and explore the multi-modal relations(i.e., the object,
audio, and question) in terms of feature interaction and model optimization.
For the former, we present an end-to-end object-oriented network that adopts a
question-conditioned clue discovery module to concentrate audio/visual
modalities on respective keywords of the question and designs a
modality-conditioned clue collection module to highlight closely associated
audio segments or visual objects. For model optimization, we propose an
object-aware adaptive-positivity learning strategy that selects the highly
semantic-matched multi-modal pair as positivity. Specifically, we design two
object-aware contrastive loss functions to identify the highly relevant
question-object pairs and audio-object pairs, respectively. These selected
pairs are constrained to have larger similarity values than the mismatched
pairs. The positivity-selecting process is adaptive as the positivity pairs
selected in each video frame may be different. These two object-aware
objectives help the model understand which objects are exactly relevant to the
question and which are making sounds. Extensive experiments on the MUSIC-AVQA
dataset demonstrate the proposed method is effective in finding favorable
audio-visual clues and also achieves new state-of-the-art question-answering
performance.

基于音频 - 视觉问题回答任务（AVQA），本文提出了一种针对细粒度视觉对象的端到端对象导向网络，通过特征交互和模型优化来探索多模态关系，并在模型优化方面提出了一种对象感知自适应正性学习策略，以选择高度语义匹配的多模态对作为正性。通过对 MUSIC-AVQA 数据集进行的大量实验证明，该方法能够有效地找到有利的音频 - 视觉线索，并取得了新的最先进的问题回答性能。