The traditional image captioning task uses generic reference captions to
provide textual information about images. Different user populations, however,
will care about different visual aspects of images. In this paper, we propose a
new task, Captioning with a Purpose (CapWAP). Our goal is to develop systems
that can be tailored to be useful for the information needs of an intended
population, rather than merely provide generic information about an image. In
this task, we use question-answer (QA) pairs---a natural expression of
information need---from users, instead of reference captions, for both training
and post-inference evaluation. We show that it is possible to use reinforcement
learning to directly optimize for the intended information need, by rewarding
outputs that allow a question answering model to provide correct answers to
sampled user questions. We convert several visual question answering datasets
into CapWAP datasets, and demonstrate that under a variety of scenarios our
purposeful captioning system learns to anticipate and fulfill specific
information needs better than its generic counterparts, as measured by QA
performance on user questions from unseen images, when using the caption alone
as context.

本文提出了一种叫做 “Captioning with a Purpose” 的新任务，使用问题 - 回答对代替传统的参考字幕进行训练和后推理评估，并使用强化学习直接优化旨在满足用户信息需求的结果。这种目的性字幕系统在使用字幕作为上下文时，能够比其通用对应物更好地预测和满足特定信息需求，在多种情况下演示了这一定论的有效性。