Audio-Visual Question Answering (AVQA) task aims to answer questions about
different visual objects, sounds, and their associations in videos. Such
naturally multi-modal videos are composed of rich and complex dynamic
audio-visual components, where most of which could be unrelated to the given
questions, or even play as interference in answering the content of interest.
Oppositely, only focusing on the question-aware audio-visual content could get
rid of influence, meanwhile enabling the model to answer more efficiently. In
this paper, we propose a Progressive Spatio-Temporal Perception Network
(PSTP-Net), which contains three modules that progressively identify key
spatio-temporal regions w.r.t. questions. Specifically, a temporal segment
selection module is first introduced to select the most relevant audio-visual
segments related to the given question. Then, a spatial region selection module
is utilized to choose the most relevant regions associated with the question
from the selected temporal segments. To further refine the selection of
features, an audio-guided visual attention module is employed to perceive the
association between auido and selected spatial regions. Finally, the
spatio-temporal features from these modules are integrated for answering the
question. Extensive experimental results on the public MUSIC-AVQA and AVQA
datasets provide compelling evidence of the effectiveness and efficiency of
PSTP-Net. Code is available at:
\href{https://github.com/GeWu-Lab/PSTP-Net}{this https URL}

为了更高效地回答关于视频中的视觉对象、声音及其关联的问题，本文提出了一种渐进式时空感知网络（PSTP-Net），通过三个模块逐步识别问题相关的关键时空区域。从公共数据集 MUSIC-AVQA 和 AVQA 的广泛实验结果来看，PSTP-Net 在有效性和效率上具备了令人信服的证据。