Learning to fuse vision and language information and representing them is an
important research problem with many applications. Recent progresses have
leveraged the ideas of pre-training (from language modeling) and attention
layers in Transformers to learn representation from datasets containing images
aligned with linguistic expressions that describe the images. In this paper, we
propose learning representations from a set of implied, visually grounded
expressions between image and text, automatically mined from those datasets. In
particular, we use denotation graphs to represent how specific concepts (such
as sentences describing images) can be linked to abstract and generic concepts
(such as short phrases) that are also visually grounded. This type of
generic-to-specific relations can be discovered using linguistic analysis
tools. We propose methods to incorporate such relations into learning
representation. We show that state-of-the-art multimodal learning models can be
further improved by leveraging automatically harvested structural relations.
The representations lead to stronger empirical results on downstream tasks of
cross-modal image retrieval, referring expression, and compositional
attribute-object recognition. Both our codes and the extracted denotation
graphs on the Flickr30K and the COCO datasets are publically available on
this https URL

本文提出利用暗示的视觉引导表达学习表示，自动地从图像和文本的数据集中挖掘出的结构关系，用于多模态学习模型中的视觉语言融合任务，证明了该方法在跨模态图片检索、指代表达和组合属性对象识别中的有效性。