Understanding realistic visual scene images together with language
descriptions is a fundamental task towards generic visual understanding.
Previous works have shown compelling comprehensive results by building
hierarchical structures for visual scenes (e.g., scene graphs) and natural
languages (e.g., dependency trees), individually. However, how to construct a
joint vision-language (VL) structure has barely been investigated. More
challenging but worthwhile, we introduce a new task that targets on inducing
such a joint VL structure in an unsupervised manner. Our goal is to bridge the
visual scene graphs and linguistic dependency trees seamlessly. Due to the lack
of VL structural data, we start by building a new dataset VLParse. Rather than
using labor-intensive labeling from scratch, we propose an automatic alignment
procedure to produce coarse structures followed by human refinement to produce
high-quality ones. Moreover, we benchmark our dataset by proposing a
contrastive learning (CL)-based framework VLGAE, short for Vision-Language
Graph Autoencoder. Our model obtains superior performance on two derived tasks,
i.e., language grammar induction and VL phrase grounding. Ablations show the
effectiveness of both visual cues and dependency relationships on fine-grained
VL structure construction.

本文提出了一种对视觉场景图和语言依赖树进行联合建模的无监督学习任务，并构造了一个新的数据集 VLParse，提出了基于对比学习的 VLGAE 框架用于 VL 短语理解和语言语法归纳。实验结果表明了视觉信息和语言依赖关系对于 VL 结构建模的有效性。