State-of-the-art vision-language models (VLMs) still have limited performance
in structural knowledge extraction, such as relations between objects. In this
work, we present ViStruct, a training framework to learn VLMs for effective
visual structural knowledge extraction. Two novel designs are incorporated.
First, we propose to leverage the inherent structure of programming language to
depict visual structural information. This approach enables explicit and
consistent representation of visual structural information of multiple
granularities, such as concepts, relations, and events, in a well-organized
structured format. Second, we introduce curriculum-based learning for VLMs to
progressively comprehend visual structures, from fundamental visual concepts to
intricate event structures. Our intuition is that lower-level knowledge may
contribute to complex visual structure understanding. Furthermore, we compile
and release a collection of datasets tailored for visual structural knowledge
extraction. We adopt a weakly-supervised approach to directly generate visual
event structures from captions for ViStruct training, capitalizing on abundant
image-caption pairs from the web. In experiments, we evaluate ViStruct on
visual structure prediction tasks, demonstrating its effectiveness in improving
the understanding of visual structures. The code is public at
https://github.com/Yangyi-Chen/vi-struct.

在这项研究中，我们提出了 ViStruct，这是一个用于学习视觉结构知识提取的训练框架，它通过两个新颖的设计元素来改进最新的视觉 - 语言模型（VLMs）在结构化知识抽取方面的性能限制。