Dialog systems need to understand dynamic visual scenes in order to have
conversations with users about the objects and events around them. Scene-aware
dialog systems for real-world applications could be developed by integrating
state-of-the-art technologies from multiple research areas, including:
end-to-end dialog technologies, which generate system responses using models
trained from dialog data; visual question answering (VQA) technologies, which
answer questions about images using learned image features; and video
description technologies, in which descriptions/captions are generated from
videos using multimodal information. We introduce a new dataset of dialogs
about videos of human behaviors. Each dialog is a typed conversation that
consists of a sequence of 10 question-and-answer(QA) pairs between two Amazon
Mechanical Turk (AMT) workers. In total, we collected dialogs on roughly 9,000
videos. Using this new dataset for Audio Visual Scene-aware dialog (AVSD), we
trained an end-to-end conversation model that generates responses in a dialog
about a video. Our experiments demonstrate that using multimodal features that
were developed for multimodal attention-based video description enhances the
quality of generated dialog about dynamic scenes (videos). Our dataset, model
code and pretrained models will be publicly available for a new Video
Scene-Aware Dialog challenge.

该论文介绍了一种新的视频场景感知对话系统，该系统将多个研究领域的最新技术整合应用 ，包括端到端的对话技术、视觉问答技术，以及视频描述技术。通过收集一个有关人类行为视频的对话数据集，作者们使用该数据集训练出一种多模态对话模型，它可以在对视频进行讨论时生成响应。最终实验结果表明，使用为多模态注意力视频描述开发的多模态特征可以提高对于动态场景（视频）生成对话的质量。