Establishing an automatic evaluation metric that closely aligns with human
judgments is essential for effectively developing image captioning models.
Recent data-driven metrics have demonstrated a stronger correlation with human
judgments than classic metrics such as CIDEr; however they lack sufficient
capabilities to handle hallucinations and generalize across diverse images and
texts partially because they compute scalar similarities merely using
embeddings learned from tasks unrelated to image captioning evaluation. In this
study, we propose Polos, a supervised automatic evaluation metric for image
captioning models. Polos computes scores from multimodal inputs, using a
parallel feature extraction mechanism that leverages embeddings trained through
large-scale contrastive learning. To train Polos, we introduce Multimodal
Metric Learning from Human Feedback (M$^2$LHF), a framework for developing
metrics based on human feedback. We constructed the Polaris dataset, which
comprises 131K human judgments from 550 evaluators, which is approximately ten
times larger than standard datasets. Our approach achieved state-of-the-art
performance on Composite, Flickr8K-Expert, Flickr8K-CF, PASCAL-50S, FOIL, and
the Polaris dataset, thereby demonstrating its effectiveness and robustness.

提出了一种基于人类反馈训练的 Polos 的自动评估度量标准用于图像字幕模型，该度量标准从多模态输入中计算得分，利用通过大规模对比学习训练的嵌入进行平行特征提取机制。通过在 Polis 数据集上取得的最新成果，证明了该方法具有效果和鲁棒性。