Neural image/video captioning models can generate accurate descriptions, but
their internal process of mapping regions to words is a black box and therefore
difficult to explain. Top-down neural saliency methods can find important
regions given a high-level semantic task such as object classification, but
cannot use a natural language sentence as the top-down input for the task. In
this paper, we propose Caption-Guided Visual Saliency to expose the
region-to-word mapping in modern encoder-decoder networks and demonstrate that
it is learned implicitly from caption training data, without any pixel-level
annotations. Our approach can produce spatial or spatiotemporal heatmaps for
both predicted captions, and for arbitrary query sentences. It recovers
saliency without the overhead of introducing explicit attention layers, and can
be used to analyze a variety of existing model architectures and improve their
design. Evaluation on large-scale video and image datasets demonstrates that
our approach achieves comparable captioning performance with existing methods
while providing more accurate saliency heatmaps. Our code is available at
visionlearninggroup.github.io/caption-guided-saliency/.

本文提出了基于标题引导的视觉显著性方法，通过训练数据隐式的学习到现代编码器解码器网络中的区域到单词的映射，而不引入显式的注意力层，从而分析各种现有模型结构，并提供更准确的显著性热图。