Understanding audio-visual content and the ability to have an informative
conversation about it have both been challenging areas for intelligent systems.
The Audio Visual Scene-aware Dialog (AVSD) challenge, organized as a track of
the Dialog System Technology Challenge 7 (DSTC7), proposes a combined task,
where a system has to answer questions pertaining to a video given a dialogue
with previous question-answer pairs and the video itself. We propose for this
task a hierarchical encoder-decoder model which computes a multi-modal
embedding of the dialogue context. It first embeds the dialogue history using
two LSTMs. We extract video and audio frames at regular intervals and compute
semantic features using pre-trained I3D and VGGish models, respectively. Before
summarizing both modalities into fixed-length vectors using LSTMs, we use FiLM
blocks to condition them on the embeddings of the current question, which
allows us to reduce the dimensionality considerably. Finally, we use an LSTM
decoder that we train with scheduled sampling and evaluate using beam search.
Compared to the modality-fusing baseline model released by the AVSD challenge
organizers, our model achieves a relative improvements of more than 16%,
scoring 0.36 BLEU-4 and more than 33%, scoring 0.997 CIDEr.

通过 AVSD 挑战，本文提出了一种层次化编码 - 解码模型来回答有关视频的问题，采用预训练的 I3D 和 VGGish 模型计算视频及音频帧的语义特征，并使用 FiLM 块进行条件编码以降低维数，最终使用 LSTM 解码器通过计划采样进行训练和束搜索进行评估，相比于 AVSD 挑战组织者发布的模态融合基准模型，我们的模型实现了超过 16％的相对改善，得分为 0.36 BLEU -4，超过 33％的得分为 0.997 CIDEr。