Generating multi-sentence descriptions for videos is one of the most challenging captioning tasks due to its high requirements for not only visual relevance but also discourse-based coherence across the sentences in the paragraph. Towards this goal, we propose a new approach called Memory-Augmented Recurrent Transformer (MART), which uses a memory module to augment the transformer architecture. The memory module generates a highly summarized memory state from the video segments and the sentence history so as to help better prediction of the next sentence (w.r.t. coreference and repetition aspects), thus encouraging coherent paragraph generation. Extensive experiments, human evaluations, and qualitative analyses on two popular datasets ActivityNet Captions and YouCookII show that MART generates more coherent and less repetitive paragraph captions than baseline methods, while maintaining relevance to the input video events. All code is available open-source at: https://github.com/jayleicn/recurrent-transformer

我们提出了一种新的方法Memory-Augmented Recurrent Transformer (MART)，它使用记忆模块来增强Transformer的结构，以更好地预测下一句话（关于指代和重复），从而鼓励连贯的段落生成。实验证明，MART生成的段落字幕比基线方法更连贯，更少重复，并保持与输入视频事件的相关性。

记忆增强循环变压器用于连贯视频段落字幕生成