Visual Dialogue task requires an agent to be engaged in a conversation with human about an image. The ability of generating detailed and non-repetitive responses is crucial for the agent to achieve human-like conversation. In this paper, we propose a novel generative decoding architecture to generate high-quality responses, which moves away from decoding the whole encoded semantics towards the design that advocates both transparency and flexibility. In this architecture, word generation is decomposed into a series of attention-based information selection steps, performed by the novel recurrent Deliberation, Abandon and Memory (DAM) module. Each DAM module performs an adaptive combination of the response-level semantics captured from the encoder and the word-level semantics specifically selected for generating each word. Therefore, the responses contain more detailed and non-repetitive descriptions while maintaining the semantic accuracy. Furthermore, DAM is flexible to cooperate with existing visual dialogue encoders and adaptive to the encoder structures by constraining the information selection mode in DAM. We apply DAM to three typical encoders and verify the performance on the VisDial v1.0 dataset. Experimental results show that the proposed models achieve new state-of-the-art performance with high-quality responses. The code is available at https://github.com/JXZe/DAM.

本文提出了一种新的生成解码结构，其中单词的生成被分解成一系列基于注意力的信息选择步骤。利用 Deliberation、Abandon 和 Memory 模块来进行适应性选择，结合了编码器捕获的响应级语义和为生成每个单词专门选择的单词级语义。该结构在可维持语义准确性的同时，生成的响应更加详细和非重复，并且相对于编码器是灵活的。将 Deliberation、Abandon 和 Memory 模块应用到三个典型的编码器，并在 VisDial v1.0 数据集上验证了性能，实验结果表明，所提出的模型取得了高质量响应的新的最优性能。

DAM: 在视觉对话中生成详细且不重复回复的思考、放弃和记忆网络