Video description is one of the most challenging problems in vision and
language understanding due to the large variability both on the video and
language side. Models, hence, typically shortcut the difficulty in recognition
and generate plausible sentences that are based on priors but are not
necessarily grounded in the video. In this work, we explicitly link the
sentence to the evidence in the video by annotating each noun phrase in a
sentence with the corresponding bounding box in one of the frames of a video.
Our dataset, ActivityNet-Entities, augments the challenging ActivityNet
Captions dataset with 158k bounding box annotations, each grounding a noun
phrase. This allows training video description models with this data, and
importantly, evaluate how grounded or "true" such model are to the video they
describe. To generate grounded captions, we propose a novel video description
model which is able to exploit these bounding box annotations. We demonstrate
the effectiveness of our model on our dataset, but also show how it can be
applied to image description on the Flickr30k Entities dataset. We achieve
state-of-the-art performance on video description, video paragraph description,
and image description and demonstrate our generated sentences are better
grounded in the video.

该文介绍了利用视频中物体的 bounding box 将描述影片的句子与影片中的客观证据联系起来，并给出了一个能够利用 bounding box 的词语注释的视频描述模型，其在视频描述、视频段落描述和图像描述上均表现出了最先进的性能和更好的联系性。