Existing datasets for narrative understanding often fail to represent the
complexity and uncertainty of relationships in real-life social scenarios. To
address this gap, we introduce a new benchmark, Conan, designed for extracting
and analysing intricate character relation graphs from detective narratives.
Specifically, we designed hierarchical relationship categories and manually
extracted and annotated role-oriented relationships from the perspectives of
various characters, incorporating both public relationships known to most
characters and secret ones known to only a few. Our experiments with advanced
Large Language Models (LLMs) like GPT-3.5, GPT-4, and Llama2 reveal their
limitations in inferencing complex relationships and handling longer
narratives. The combination of the Conan dataset and our pipeline strategy is
geared towards understanding the ability of LLMs to comprehend nuanced
relational dynamics in narrative contexts.

为了解决现有叙述理解数据集无法表示现实社交情景中关系的复杂性和不确定性的问题，我们引入了一个新的基准，Conan，用于从侦探叙述中提取和分析错综复杂的角色关系图。我们设计了层次关系类别，并从各个角色的角度手动提取和注释了面向角色的关系，包括大多数角色所知的公开关系和只有少数人知道的秘密关系。我们使用 GPT-3.5、GPT-4 和 Llama2 等先进的大型语言模型进行实验，揭示了它们在推理复杂关系和处理较长叙述方面的局限性。Conan 数据集和我们的流程策略的结合旨在验证大型语言模型在理解叙述情境中微妙关系动态的能力。