In this paper, we present our solution to the New frontiers for Zero-shot
Image Captioning Challenge. Different from the traditional image captioning
datasets, this challenge includes a larger new variety of visual concepts from
many domains (such as COVID-19) as well as various image types (photographs,
illustrations, graphics). For the data level, we collect external training data
from Laion-5B, a large-scale CLIP-filtered image-text dataset. For the model
level, we use OFA, a large-scale visual-language pre-training model based on
handcrafted templates, to perform the image captioning task. In addition, we
introduce contrastive learning to align image-text pairs to learn new visual
concepts in the pre-training stage. Then, we propose a similarity-bucket
strategy and incorporate this strategy into the template to force the model to
generate higher quality and more matching captions. Finally, by
retrieval-augmented strategy, we construct a content-rich template, containing
the most relevant top-k captions from other image-text pairs, to guide the
model in generating semantic-rich captions. Our method ranks first on the
leaderboard, achieving 105.17 and 325.72 Cider-Score in the validation and test
phase, respectively.

我们在新的零样本图像字幕挑战中提出了解决方案，通过使用 Laion-5B 数据集进行数据级别和 OFA 模型进行模型级别的图像字幕任务，结合对齐图像文本对的对比学习和相似性桶策略，以及检索增强策略构建内容丰富的模板，最终在排行榜上取得了第一名的成绩。