Video paragraph captioning aims to describe multiple events in untrimmed
videos with descriptive paragraphs. Existing approaches mainly solve the
problem in two steps: event detection and then event captioning. Such two-step
manner makes the quality of generated paragraphs highly dependent on the
accuracy of event proposal detection which is already a challenging task. In
this paper, we propose a paragraph captioning model which eschews the
problematic event detection stage and directly generates paragraphs for
untrimmed videos. To describe coherent and diverse events, we propose to
enhance the conventional temporal attention with dynamic video memories, which
progressively exposes new video features and suppresses over-accessed video
contents to control visual focuses of the model. In addition, a
diversity-driven training strategy is proposed to improve diversity of
paragraph on the language perspective. Considering that untrimmed videos
generally contain massive but redundant frames, we further augment the video
encoder with keyframe awareness to improve efficiency. Experimental results on
the ActivityNet and Charades datasets show that our proposed model
significantly outperforms the state-of-the-art performance on both accuracy and
diversity metrics without using any event boundary annotations. Code will be
released at this https URL

本研究提出了一种基于自适应动态视频记忆的视频段落字幕生成模型，并采用多样性驱动的训练策略，通过关注模型的视觉焦点和提高多样性来生成具有连贯性和多样性特征的段落。实验证明该模型不但提高了生成准确度和多样性，而且还能有效地处理未剪辑的视频数据。

面向未剪辑视频的多样段落字幕

Towards Diverse Paragraph Captioning for Untrimmed Videos

Recent studies on interpretability of attention distributions have led to
notions of faithful and plausible explanations for a model's predictions.
Attention distributions can be considered a faithful explanation if a higher
attention weight implies a greater impact on the model's prediction. They can
be considered a plausible explanation if they provide a human-understandable
justification for the model's predictions. In this work, we first explain why
current attention mechanisms in LSTM based encoders can neither provide a
faithful nor a plausible explanation of the model's predictions. We observe
that in LSTM based encoders the hidden representations at different time-steps
are very similar to each other (high conicity) and attention weights in these
situations do not carry much meaning because even a random permutation of the
attention weights does not affect the model's predictions. Based on experiments
on a wide variety of tasks and datasets, we observe attention distributions
often attribute the model's predictions to unimportant words such as
punctuation and fail to offer a plausible explanation for the predictions. To
make attention mechanisms more faithful and plausible, we propose a modified
LSTM cell with a diversity-driven training objective that ensures that the
hidden representations learned at different time steps are diverse. We show
that the resulting attention distributions offer more transparency as they (i)
provide a more precise importance ranking of the hidden states (ii) are better
indicative of words important for the model's predictions (iii) correlate
better with gradient-based attribution methods. Human evaluations indicate that
the attention distributions learned by our model offer a plausible explanation
of the model's predictions. Our code has been made publicly available at
this https URL

本文研究基于 LSTM 的编码器的注意力机制的可解释性问题，发现现有的注意力机制不能提供清晰的解释，因为同步隐藏状态之间的高相关性使得注意权重缺乏了传达重要信息的能力。为了使注意力机制更为可信和可解释，作者提出一种基于多样性驱动的训练目标修改 LSTM 结构，实现不同时步的隐藏状态的多样化，以提高注意力分布的质量和可解释性。