We propose a simple yet effective and robust method for contrastive
captioning: generating discriminative captions that distinguish target images
from very similar alternative distractor images. Our approach is built on a
pragmatic inference procedure that formulates captioning as a reference game
between a speaker, which produces possible captions describing the target, and
a listener, which selects the target given the caption. Unlike previous methods
that derive both speaker and listener distributions from a single captioning
model, we leverage an off-the-shelf CLIP model to parameterize the listener.
Compared with captioner-only pragmatic models, our method benefits from rich
vision language alignment representations from CLIP when reasoning over
distractors. Like previous methods for discriminative captioning, our method
uses a hyperparameter to control the tradeoff between the informativity (how
likely captions are to allow a human listener to discriminate the target image)
and the fluency of the captions. However, we find that our method is
substantially more robust to the value of this hyperparameter than past
methods, which allows us to automatically optimize the captions for
informativity - outperforming past methods for discriminative captioning by 11%
to 15% accuracy in human evaluations

本文提出了一种基于对比度创建区分性字幕的方法，该方法建立在将字幕作为参照游戏进行的实用推理程序上，并利用现成的 CLIP 模型对接收者进行参数化，以在推理分别对待种类相似的其他干扰图像时从 CLIP 获得视觉 - 语言对齐表示，其自动优化字幕的信息性，优越于人们对具有区分性字幕的图像的准确性评估。