In scholarly documents, figures provide a straightforward way of
communicating scientific findings to readers. Automating figure caption
generation helps move model understandings of scientific documents beyond text
and will help authors write informative captions that facilitate communicating
scientific findings. Unlike previous studies, we reframe scientific figure
captioning as a knowledge-augmented image captioning task that models need to
utilize knowledge embedded across modalities for caption generation. To this
end, we extended the large-scale SciCap
dataset~\cite{hsu-etal-2021-scicap-generating} to SciCap+ which includes
mention-paragraphs (paragraphs mentioning figures) and OCR tokens. Then, we
conduct experiments with the M4C-Captioner (a multimodal transformer-based
model with a pointer network) as a baseline for our study. Our results indicate
that mention-paragraphs serves as additional context knowledge, which
significantly boosts the automatic standard image caption evaluation scores
compared to the figure-only baselines. Human evaluations further reveal the
challenges of generating figure captions that are informative to readers. The
code and SciCap+ dataset will be publicly available at
this https URL

通过扩充 SciCap 数据集，创新性地将科技论文中的图例标注生成作为知识增强图像标注任务，提高自动标注的效率与准确性，并探讨了从不同模视觉图像和文本知识融合的角度进行标注生成的实验研究。

SciCap+: 研究科学图标题挑战的知识增强数据集

SciCap+: A Knowledge Augmented Dataset to Study the Challenges of  Scientific Figure Captioning

Visual Relation Detection (VRD) aims to detect relationships between objects
for image understanding. Most existing VRD methods rely on thousands of
training samples of each relationship to achieve satisfactory performance. Some
recent papers tackle this problem by few-shot learning with elaborately
designed pipelines and pre-trained word vectors. However, the performance of
existing few-shot VRD models is severely hampered by the poor generalization
capability, as they struggle to handle the vast semantic diversity of visual
relationships. Nonetheless, humans have the ability to learn new relationships
with just few examples based on their knowledge. Inspired by this, we devise a
knowledge-augmented, few-shot VRD framework leveraging both textual knowledge
and visual relation knowledge to improve the generalization ability of few-shot
VRD. The textual knowledge and visual relation knowledge are acquired from a
pre-trained language model and an automatically constructed visual relation
knowledge graph, respectively. We extensively validate the effectiveness of our
framework. Experiments conducted on three benchmarks from the commonly used
Visual Genome dataset show that our performance surpasses existing
state-of-the-art models with a large improvement.

本研究提出了一个基于知识增强的少样本学习（few-shot learning）框架，利用预训练语言模型和自动构建的视觉关系知识图谱获取了文本知识和视觉关系知识，从而提高了少样本学习的实用性和性能。与现有最先进模型相比，在常用的 Visual Genome 数据集的三个基准测试中，本方法的性能大幅超越了现有最先进模型。