Foundation Models (FMs) have demonstrated remarkable insights into the
relational dynamics of the world, leading to the crucial question: how do these
models acquire an understanding of world hybrid relations? Traditional
statistical learning, particularly for prediction problems, may overlook the
rich and inherently structured information from the data, especially regarding
the relationships between objects. We introduce a mathematical model that
formalizes relational learning as hypergraph recovery to study pre-training of
FMs. In our framework, the world is represented as a hypergraph, with data
abstracted as random samples from hyperedges. We theoretically examine the
feasibility of a Pre-Trained Model (PTM) to recover this hypergraph and analyze
the data efficiency in a minimax near-optimal style. By integrating rich graph
theories into the realm of PTMs, our mathematical framework offers powerful
tools for an in-depth understanding of pre-training from a unique perspective
and can be used under various scenarios. As an example, we extend the framework
to entity alignment in multimodal learning.

基础模型具有对世界关系动态的显著洞察力，但它们如何获得对世界混合关系的理解是一个关键问题。本研究引入了一个数学模型，将关系学习形式化为超图恢复，以研究基础模型的预训练。我们的框架将世界表示为一个超图，数据抽象为超边的随机样本。通过将丰富的图论集成到预训练模型的领域中，我们的数学框架提供了强大的工具，以独特的视角深入理解预训练，并可在各种场景下使用。作为示例，我们将该框架扩展到多模式学习中的实体对齐问题。

预训练模型中的关系学习：来自超图恢复视角的理论

Relational Learning in Pre-Trained Models: A Theory from Hypergraph  Recovery Perspective

Existing datasets for narrative understanding often fail to represent the
complexity and uncertainty of relationships in real-life social scenarios. To
address this gap, we introduce a new benchmark, Conan, designed for extracting
and analysing intricate character relation graphs from detective narratives.
Specifically, we designed hierarchical relationship categories and manually
extracted and annotated role-oriented relationships from the perspectives of
various characters, incorporating both public relationships known to most
characters and secret ones known to only a few. Our experiments with advanced
Large Language Models (LLMs) like GPT-3.5, GPT-4, and Llama2 reveal their
limitations in inferencing complex relationships and handling longer
narratives. The combination of the Conan dataset and our pipeline strategy is
geared towards understanding the ability of LLMs to comprehend nuanced
relational dynamics in narrative contexts.

为了解决现有叙述理解数据集无法表示现实社交情景中关系的复杂性和不确定性的问题，我们引入了一个新的基准，Conan，用于从侦探叙述中提取和分析错综复杂的角色关系图。我们设计了层次关系类别，并从各个角色的角度手动提取和注释了面向角色的关系，包括大多数角色所知的公开关系和只有少数人知道的秘密关系。我们使用 GPT-3.5、GPT-4 和 Llama2 等先进的大型语言模型进行实验，揭示了它们在推理复杂关系和处理较长叙述方面的局限性。Conan 数据集和我们的流程策略的结合旨在验证大型语言模型在理解叙述情境中微妙关系动态的能力。

大型语言模型的不足：探察叙事中的复杂关系理解

Large Language Models Fall Short: Understanding Complex Relationships in  Detective Narratives

In this paper, we propose a novel approach to predict group activities given
the beginning frames with incomplete activity executions. Existing action
prediction approaches learn to enhance the representation power of the partial
observation. However, for group activity prediction, the relation evolution of
people's activity and their positions over time is an important cue for
predicting group activity. To this end, we propose a sequential relational
anticipation model (SRAM) that summarizes the relational dynamics in the
partial observation and progressively anticipates the group representations
with rich discriminative information. Our model explicitly anticipates both
activity features and positions by two graph auto-encoders, aiming to learn a
discriminative group representation for group activity prediction. Experimental
results on two popularly used datasets demonstrate that our approach
significantly outperforms the state-of-the-art activity prediction methods.

本研究提出一种基于序列关系预测模型的组活动预测方法，该模型可以总结部分观测数据中的关系动态并逐步预测具有区分性信息的群体表示，通过两种图形自编码器预测活动特征和位置，并在实验中取得了显著的预测效果。