There is considerable interest in the task of automatically generating image
captions. However, evaluation is challenging. Existing automatic evaluation
metrics are primarily sensitive to n-gram overlap, which is neither necessary
nor sufficient for the task of simulating human judgment. We hypothesize that
semantic propositional content is an important component of human caption
evaluation, and propose a new automated caption evaluation metric defined over
scene graphs coined SPICE. Extensive evaluations across a range of models and
datasets indicate that SPICE captures human judgments over model-generated
captions better than other automatic metrics (e.g., system-level correlation of
0.88 with human judgments on the MS COCO dataset, versus 0.43 for CIDEr and
0.53 for METEOR). Furthermore, SPICE can answer questions such as `which
caption-generator best understands colors?' and `can caption-generators count?'

提出了基于场景图谱的自动生成图像标题评估指标 SPICE，其捕捉人类判断模型生成标题的能力优于现有指标，并能回答哪个标题生成器更懂颜色和能否计数等问题。