Neural captioners are typically trained to mimic human-generated references
without optimizing for any specific communication goal, leading to problems
such as the generation of vague captions. In this paper, we show that
fine-tuning an out-of-the-box neural captioner with a self-supervised
discriminative communication objective helps to recover a plain, visually
descriptive language that is more informative about image contents. Given a
target image, the system must learn to produce a description that enables an
out-of-the-box text-conditioned image retriever to identify such image among a
set of candidates. We experiment with the popular ClipCap captioner, also
replicating the main results with BLIP. In terms of similarity to ground-truth
human descriptions, the captions emerging from discriminative finetuning lag
slightly behind those generated by the non-finetuned model, when the latter is
trained and tested on the same caption dataset. However, when the model is used
without further tuning to generate captions for out-of-domain datasets, our
discriminatively-finetuned captioner generates descriptions that resemble human
references more than those produced by the same captioner without finetuning.
We further show that, on the Conceptual Captions dataset, discriminatively
finetuned captions are more helpful than either vanilla ClipCap captions or
ground-truth captions for human annotators tasked with an image discrimination
task.

本文提出在自我监督的证明性沟通目标情况下对预先训练的神经字幕系统进行微调，使其生成更详细的图像描述，并在 Conceptual Captions 数据集上进行验证。