Recent image captioning models are achieving impressive results based on
popular metrics, i.e., BLEU, CIDEr, and SPICE. However, focusing on the most
popular metrics that only consider the overlap between the generated captions
and human annotation could result in using common words and phrases, which
lacks distinctiveness, i.e., many similar images have the same caption. In this
paper, we aim to improve the distinctiveness of image captions via comparing
and reweighting with a set of similar images. First, we propose a
distinctiveness metric -- between-set CIDEr (CIDErBtw) to evaluate the
distinctiveness of a caption with respect to those of similar images. Our
metric reveals that the human annotations of each image in the MSCOCO dataset
are not equivalent based on distinctiveness; however, previous works normally
treat the human annotations equally during training, which could be a reason
for generating less distinctive captions. In contrast, we reweight each
ground-truth caption according to its distinctiveness during training. We
further integrate a long-tailed weight strategy to highlight the rare words
that contain more information, and captions from the similar image set are
sampled as negative examples to encourage the generated sentence to be unique.
Finally, extensive experiments are conducted, showing that our proposed
approach significantly improves both distinctiveness (as measured by CIDErBtw
and retrieval metrics) and accuracy (e.g., as measured by CIDEr) for a wide
variety of image captioning baselines. These results are further confirmed
through a user study.

本研究旨在通过比较和重新加权计算一组相似图像来提高图像字幕的独特性。我们提出了一种称为 CIDErBtw 的独特性度量，用于衡量字幕与类似图像的独特性，并在训练期间重新加权了每个人工注释，以提高生成的字幕的独特性和准确性。