In this paper, we introduce Key-Value Memory Networks to a multimodal setting
and a novel key-addressing mechanism to deal with sequence-to-sequence models.
The proposed model naturally decomposes the problem of video captioning into
vision and language segments, dealing with them as key-value pairs. More
specifically, we learn a semantic embedding (v) corresponding to each frame (k)
in the video, thereby creating (k, v) memory slots. We propose to find the next
step attention weights conditioned on the previous attention distributions for
the key-value memory slots in the memory addressing schema. Exploiting this
flexibility of the framework, we additionally capture spatial dependencies
while mapping from the visual to semantic embedding. Experiments done on the
Youtube2Text dataset demonstrate usefulness of recurrent key-addressing, while
achieving competitive scores on BLEU@4, METEOR metrics against state-of-the-art
models.

本文提出了 Key-Value Memory Networks 应用于多模态设置的方法，以及一种新的键寻址机制，将视频字幕生成问题自然地分解为视觉和语言端，将其作为键 - 值对处理，并在寻址模式下提出了一种递归关注的方法来捕捉语境信息，通过实验发现，这种方法可以提高 BLEU@4，METEOR 得分，并实现了与最先进方法竞争性能。