In scholarly documents, figures provide a straightforward way of
communicating scientific findings to readers. Automating figure caption
generation helps move model understandings of scientific documents beyond text
and will help authors write informative captions that facilitate communicating
scientific findings. Unlike previous studies, we reframe scientific figure
captioning as a knowledge-augmented image captioning task that models need to
utilize knowledge embedded across modalities for caption generation. To this
end, we extended the large-scale SciCap
dataset~\cite{hsu-etal-2021-scicap-generating} to SciCap+ which includes
mention-paragraphs (paragraphs mentioning figures) and OCR tokens. Then, we
conduct experiments with the M4C-Captioner (a multimodal transformer-based
model with a pointer network) as a baseline for our study. Our results indicate
that mention-paragraphs serves as additional context knowledge, which
significantly boosts the automatic standard image caption evaluation scores
compared to the figure-only baselines. Human evaluations further reveal the
challenges of generating figure captions that are informative to readers. The
code and SciCap+ dataset will be publicly available at
this https URL

通过扩充 SciCap 数据集，创新性地将科技论文中的图例标注生成作为知识增强图像标注任务，提高自动标注的效率与准确性，并探讨了从不同模视觉图像和文本知识融合的角度进行标注生成的实验研究。