Developing Video-Grounded Dialogue Systems (VGDS), where a dialogue is
conducted based on visual and audio aspects of a given video, is significantly
more challenging than traditional image or text-grounded dialogue systems
because (1) feature space of videos span across multiple picture frames, making
it difficult to obtain semantic information; and (2) a dialogue agent must
perceive and process information from different modalities (audio, video,
caption, etc.) to obtain a comprehensive understanding. Most existing work is
based on RNNs and sequence-to-sequence architectures, which are not very
effective for capturing complex long-term dependencies (like in videos). To
overcome this, we propose Multimodal Transformer Networks (MTN) to encode
videos and incorporate information from different modalities. We also propose
query-aware attention through an auto-encoder to extract query-aware features
from non-text modalities. We develop a training procedure to simulate
token-level decoding to improve the quality of generated responses during
inference. We get state of the art performance on Dialogue System Technology
Challenge 7 (DSTC7). Our model also generalizes to another multimodal
visual-grounded dialogue task, and obtains promising performance. We
implemented our models using PyTorch and the code is released at
this https URL

本研究提出了一种使用多模态转换网络（MTN）来编码和整合不同模态信息的视频对话系统。在对话生成过程中，我们提出了一个训练程序来模拟标记级解码，从而提高了生成响应的质量。我们在 DSTC7 上获得了最优秀的表现，并且我们的模型具有很好的泛化性能。