State-of-the-art vision-language models (VLMs) still have limited performance
in structural knowledge extraction, such as relations between objects. In this
work, we present ViStruct, a training framework to learn VLMs for effective
visual structural knowledge extraction. Two novel designs are incorporated.
First, we propose to leverage the inherent structure of programming language to
depict visual structural information. This approach enables explicit and
consistent representation of visual structural information of multiple
granularities, such as concepts, relations, and events, in a well-organized
structured format. Second, we introduce curriculum-based learning for VLMs to
progressively comprehend visual structures, from fundamental visual concepts to
intricate event structures. Our intuition is that lower-level knowledge may
contribute to complex visual structure understanding. Furthermore, we compile
and release a collection of datasets tailored for visual structural knowledge
extraction. We adopt a weakly-supervised approach to directly generate visual
event structures from captions for ViStruct training, capitalizing on abundant
image-caption pairs from the web. In experiments, we evaluate ViStruct on
visual structure prediction tasks, demonstrating its effectiveness in improving
the understanding of visual structures. The code is public at
https://github.com/Yangyi-Chen/vi-struct.

在这项研究中，我们提出了 ViStruct，这是一个用于学习视觉结构知识提取的训练框架，它通过两个新颖的设计元素来改进最新的视觉 - 语言模型（VLMs）在结构化知识抽取方面的性能限制。

ViStruct: 基于课程引导的代码 - 视觉表征的视觉结构知识提取

ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided  Code-Vision Representation

Unsupervised Environment Design (UED) is a paradigm for training generally
capable agents to achieve good zero-shot transfer performance. This paradigm
hinges on automatically generating a curriculum of training environments.
Leading approaches for UED predominantly use randomly generated environment
instances to train the agent. While these methods exhibit good zero-shot
transfer performance, they often encounter challenges in effectively exploring
large design spaces or leveraging previously discovered underlying structures,
To address these challenges, we introduce a novel framework based on
Hierarchical MDP (Markov Decision Processes). Our approach includes an
upper-level teacher's MDP responsible for training a lower-level MDP student
agent, guided by the student's performance. To expedite the learning of the
upper leavel MDP, we leverage recent advancements in generative modeling to
generate synthetic experience dataset for training the teacher agent. Our
algorithm, called Synthetically-enhanced Hierarchical Environment Design
(SHED), significantly reduces the resource-intensive interactions between the
agent and the environment. To validate the effectiveness of SHED, we conduct
empirical experiments across various domains, with the goal of developing an
efficient and robust agent under limited training resources. Our results show
the manifold advantages of SHED and highlight its effectiveness as a potent
instrument for curriculum-based learning within the UED framework. This work
contributes to exploring the next generation of RL agents capable of adeptly
handling an ever-expanding range of complex tasks.

无监督环境设计（UED）是一种培训通常能够实现良好零 - shot 转移性能的能力强大的代理的范式。我们提出了一种基于分层 MDP（马尔可夫决策过程）的新框架，通过指导学生的性能，上层 MDP 来培训下层 MDP 学生代理。我们的算法 SHED（Synthetically-enhanced Hierarchical Environment Design）显著减少了代理与环境之间资源密集型的交互，并证明了 SHED 的多种优势，以及它作为 UED 框架中的一种有效工具的效果。