Image captioning has received significant attention with remarkable
improvements in recent advances. Nevertheless, images in the wild encapsulate
rich knowledge and cannot be sufficiently described with models built on
image-caption pairs containing only in-domain objects. In this paper, we
propose to address the problem by augmenting standard deep captioning
architectures with object learners. Specifically, we present Long Short-Term
Memory with Pointing (LSTM-P) --- a new architecture that facilitates
vocabulary expansion and produces novel objects via pointing mechanism.
Technically, object learners are initially pre-trained on available object
recognition data. Pointing in LSTM-P then balances the probability between
generating a word through LSTM and copying a word from the recognized objects
at each time step in decoder stage. Furthermore, our captioning encourages
global coverage of objects in the sentence. Extensive experiments are conducted
on both held-out COCO image captioning and ImageNet datasets for describing
novel objects, and superior results are reported when comparing to
state-of-the-art approaches. More remarkably, we obtain an average of 60.9% in
F1 score on held-out COCO~dataset.

提出了一种新的深度图片描述模型 Long Short-Term Memory with Pointing (LSTM-P)，结合对象学习方法和指向机制实现了词汇的扩充和全局物体涵盖。在 COCO 数据集和 ImageNet 数据集上的实验结果显示其优于现有方法，最终在 COCO 数据集上获得了平均 F1 分数为 60.9% 的表现