Understanding dynamic scenes and dialogue contexts in order to converse with
users has been challenging for multimodal dialogue systems. The 8-th Dialog
System Technology Challenge (DSTC8) proposed an Audio Visual Scene-Aware Dialog
(AVSD) task, which contains multiple modalities including audio, vision, and
language, to evaluate how dialogue systems understand different modalities and
response to users. In this paper, we proposed a multi-step joint-modality
attention network (JMAN) based on recurrent neural network (RNN) to reason on
videos. Our model performs a multi-step attention mechanism and jointly
considers both visual and textual representations in each reasoning process to
better integrate information from the two different modalities. Compared to the
baseline released by AVSD organizers, our model achieves a relative 12.1% and
22.4% improvement over the baseline on ROUGE-L score and CIDEr score.

本文提出了一种基于循环神经网络的多步关注机制的多模态联合注意网络（JMAN），用于对视频进行推理，该模型在每个推理过程中联合考虑了视觉和文本表示，以更好地集成两种不同模态的信息。与 AVSD 组织发布的基线相比，我们的模型在 ROUGE-L 得分和 CIDEr 得分上相对提高了 12.1％和 22.4％。