Developing Video-Grounded Dialogue Systems (VGDS), where a dialogue is
conducted based on visual and audio aspects of a given video, is significantly
more challenging than traditional image or text-grounded dialogue systems
because (1) feature space of videos span across multiple picture frames, making
it difficult to obtain semantic information; and (2) a dialogue agent must
perceive and process information from different modalities (audio, video,
caption, etc.) to obtain a comprehensive understanding. Most existing work is
based on RNNs and sequence-to-sequence architectures, which are not very
effective for capturing complex long-term dependencies (like in videos). To
overcome this, we propose Multimodal Transformer Networks (MTN) to encode
videos and incorporate information from different modalities. We also propose
query-aware attention through an auto-encoder to extract query-aware features
from non-text modalities. We develop a training procedure to simulate
token-level decoding to improve the quality of generated responses during
inference. We get state of the art performance on Dialogue System Technology
Challenge 7 (DSTC7). Our model also generalizes to another multimodal
visual-grounded dialogue task, and obtains promising performance. We
implemented our models using PyTorch and the code is released at
this https URL

本研究提出了一种使用多模态转换网络（MTN）来编码和整合不同模态信息的视频对话系统。在对话生成过程中，我们提出了一个训练程序来模拟标记级解码，从而提高了生成响应的质量。我们在 DSTC7 上获得了最优秀的表现，并且我们的模型具有很好的泛化性能。

用于端到端视频对话系统的多模态 Transformer 网络

Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue  Systems

This paper presents an end-to-end response selection model for Track 1 of the
7th Dialogue System Technology Challenges (DSTC7). This task focuses on
selecting the correct next utterance from a set of candidates given a partial
conversation. We propose an end-to-end neural network based on enhanced
sequential inference model (ESIM) for this task. Our proposed model differs
from the original ESIM model in the following four aspects. First, a new word
representation method which combines the general pre-trained word embeddings
with those estimated on the task-specific training set is adopted in order to
address the challenge of out-of-vocabulary (OOV) words. Second, an attentive
hierarchical recurrent encoder (AHRE) is designed which is capable to encode
sentences hierarchically and generate more descriptive representations by
aggregation. Third, a new pooling method which combines multi-dimensional
pooling and last-state pooling is used instead of the simple combination of max
pooling and average pooling in the original ESIM. Last, a modification layer is
added before the softmax layer to emphasize the importance of the last
utterance in the context for response selection. In the released evaluation
results of DSTC7, our proposed method ranked second on the Ubuntu dataset and
third on the Advising dataset in subtask 1 of Track 1.

本文提出了一种基于增强的序列推理模型的端到端响应选择模型，该模型采用了新的词表示方法、带有注意力机制的分层循环编码器、多维池化和池化方法等技术来解决 DSTC7 任务中响应选择的问题，并在 DSTC7 的评估结果中获得了较高排名。