Dense video captioning is an extremely challenging task since accurate and coherent description of events in a video requires holistic understanding of video contents as well as contextual reasoning of individual events. Most existing approaches handle this problem by first detecting event proposals from a video and then captioning on a subset of the proposals. As a result, the generated sentences are prone to be redundant or inconsistent since they fail to consider temporal dependency between events. To tackle this challenge, we propose a novel dense video captioning framework, which models temporal dependency across events in a video explicitly and leverages visual and linguistic context from prior events for coherent storytelling. This objective is achieved by 1) integrating an event sequence generation network to select a sequence of event proposals adaptively, and 2) feeding the sequence of event proposals to our sequential video captioning network, which is trained by reinforcement learning with two-level rewards at both event and episode levels for better context modeling. The proposed technique achieves outstanding performances on ActivityNet Captions dataset in most metrics.

本文提出了一种新颖的密集视频字幕框架，它通过显式建模视频中事件的时间依赖性并利用先前事件的视觉和语言上下文来实现连贯的叙述。该框架由事件序列生成网络和序列视频字幕网络组成，后者利用强化学习进行训练，并在事件和剧集两个级别上进行两级奖励以实现更好的上下文建模。在大多数指标上，该方法在ActivityNet Captions数据集上取得了出色的表现。

简化密集视频字幕生成