In this paper we propose the construction of linguistic descriptions of
images. This is achieved through the extraction of scene description graphs
(SDGs) from visual scenes using an automatically constructed knowledge base.
SDGs are constructed using both vision and reasoning. Specifically, commonsense
reasoning is applied on (a) detections obtained from existing perception
methods on given images, (b) a "commonsense" knowledge base constructed using
natural language processing of image annotations and (c) lexical ontological
knowledge from resources such as WordNet. Amazon Mechanical Turk(AMT)-based
evaluations on Flickr8k, Flickr30k and MS-COCO datasets show that in most
cases, sentences auto-constructed from SDGs obtained by our method give a more
relevant and thorough description of an image than a recent state-of-the-art
image caption based approach. Our Image-Sentence Alignment Evaluation results
are also comparable to that of the recent state-of-the art approaches.

本文提出了一种利用视觉场景中的场景描述图（SDG）从图像中提取语言描述的方法，该方法通过自动构建的知识库应用视觉和推理的方式构建 SDG，并使用自然语言处理的图像注释进行的 “常识” 知识库和来自资源（如 WordNet）的词汇本体知识。通过针对 Flickr8k、Flickr30k 和 MS-COCO 数据集的基于 Amazon Mechanical Turk（AMT）的评估，我们的方法获得的自动构建句子的 SDGs 通常比基于最新的最先进的图像标题方法更相关和全面的描述了图像。我们的图像 - 句子对齐评估结果也与最新的最先进方法相当。