There is growing interest in systems that generate captions for scientific
figures. However, assessing these systems output poses a significant challenge.
Human evaluation requires academic expertise and is costly, while automatic
evaluation depends on often low-quality author-written captions. This paper
investigates using large language models (LLMs) as a cost-effective,
reference-free method for evaluating figure captions. We first constructed
SCICAP-EVAL, a human evaluation dataset that contains human judgments for 3,600
scientific figure captions, both original and machine-made, for 600 arXiv
figures. We then prompted LLMs like GPT-4 and GPT-3 to score (1-6) each caption
based on its potential to aid reader understanding, given relevant context such
as figure-mentioning paragraphs. Results show that GPT-4, used as a zero-shot
evaluator, outperformed all other models and even surpassed assessments made by
Computer Science and Informatics undergraduates, achieving a Kendall
correlation score of 0.401 with Ph.D. students rankings

使用大型语言模型（LLMs）作为一种经济的、无需参考的方法来评估科学图的标题，通过与人类学术专家评估和其他模型评估相比较，GPT-4 在评估中表现出色，甚至超过了计算机科学与信息学本科生的评估结果。

GPT-4 作为科学图注的有效零样学习评估器

GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions

Effective figure captions are crucial for clear comprehension of scientific
figures, yet poor caption writing remains a common issue in scientific
articles. Our study of arXiv cs.CL papers found that 53.88% of captions were
rated as unhelpful or worse by domain experts, showing the need for better
caption generation. Previous efforts in figure caption generation treated it as
a vision task, aimed at creating a model to understand visual content and
complex contextual information. Our findings, however, demonstrate that over
75% of figure captions' tokens align with corresponding figure-mentioning
paragraphs, indicating great potential for language technology to solve this
task. In this paper, we present a novel approach for generating figure captions
in scientific documents using text summarization techniques. Our approach
extracts sentences referencing the target figure, then summarizes them into a
concise caption. In the experiments on real-world arXiv papers (81.2% were
published at academic conferences), our method, using only text data,
outperformed previous approaches in both automatic and human evaluations. We
further conducted data-driven investigations into the two core challenges: (i)
low-quality author-written captions and (ii) the absence of a standard for good
captions. We found that our models could generate improved captions for figures
with original captions rated as unhelpful, and the model trained on captions
with more than 30 tokens produced higher-quality captions. We also found that
good captions often include the high-level takeaway of the figure. Our work
proves the effectiveness of text summarization in generating figure captions
for scholarly articles, outperforming prior vision-based approaches. Our
findings have practical implications for future figure captioning systems,
improving scientific communication clarity.

本文介绍用纯文本方法生成科学文献图注的一种新方法，使用了提取目标图像引用的句子然后将其总结为简明的图注的技术。通过真实世界 arXiv 论文的实验，我们的方法在自动和人工评估中都优于以前的方法，证明了纯文本技术在生成学术文章中的图片标注方面的有效性。