Current captioning datasets, focus on object-centric captions, describing the
visible objects in the image, often ending up stating the obvious (for humans),
e.g. "people eating food in a park". Although these datasets are useful to
evaluate the ability of Vision & Language models to recognize the visual
content, they lack in expressing trivial abstract concepts, e.g. "people having
a picnic". Such concepts are licensed by human's personal experience and
contribute to forming common sense assumptions. We present the High-Level
Dataset; a dataset extending 14997 images of the COCO dataset with 134973
human-annotated (high-level) abstract captions collected along three axes:
scenes, actions and rationales. We describe and release such dataset and we
show how it can be used to assess models' multimodal grounding of abstract
concepts and enrich models' visio-lingusitic representations. Moreover, we
describe potential tasks enabled by this dataset involving high- and low-level
concepts interactions.

本文介绍一个新的高级数据集（High-Level Dataset），可以拓展经典 COOC 数据集，使得机器学习模型更好地理解抽象概念，并进一步提升模型的多模态融合能力。