Training image captioning models using teacher forcing results in very
generic samples, whereas more distinctive captions can be very useful in
retrieval applications or to produce alternative texts describing images for
accessibility. Reinforcement Learning (RL) allows to use cross-modal retrieval
similarity score between the generated caption and the input image as reward to
guide the training, leading to more distinctive captions. Recent studies show
that pre-trained cross-modal retrieval models can be used to provide this
reward, completely eliminating the need for reference captions. However, we
argue in this paper that Ground Truth (GT) captions can still be useful in this
RL framework. We propose a new image captioning model training strategy that
makes use of GT captions in different ways. Firstly, they can be used to train
a simple MLP discriminator that serves as a regularization to prevent reward
hacking and ensures the fluency of generated captions, resulting in a textual
GAN setup extended for multimodal inputs. Secondly, they can serve as
additional trajectories in the RL strategy, resulting in a teacher forcing loss
weighted by the similarity of the GT to the image. This objective acts as an
additional learning signal grounded to the distribution of the GT captions.
Thirdly, they can serve as strong baselines when added to the pool of captions
used to compute the proposed contrastive reward to reduce the variance of
gradient estimate. Experiments on MS-COCO demonstrate the interest of the
proposed training strategy to produce highly distinctive captions while
maintaining high writing quality.

用训练图像字幕模型的教师强迫方法生成的样本非常通用，而更具有独特性的字幕对于检索应用或生成描述图像的替代文本以提高可访问性非常有用。这篇论文提出了一种新的图像字幕模型训练策略，其中利用了不同方式的真实字幕，从而在保持高写作质量的同时生成高度独特的字幕。