Recent supervised models for event coding vastly outperform pattern-matching
methods. However, their reliance solely on new annotations disregards the vast
knowledge within expert databases, hindering their applicability to
fine-grained classification. To address these limitations, we explore zero-shot
approaches for political event ontology relation classification, by leveraging
knowledge from established annotation codebooks. Our study encompasses both
ChatGPT and a novel natural language inference (NLI) based approach named ZSP.
ZSP adopts a tree-query framework that deconstructs the task into context,
modality, and class disambiguation levels. This framework improves
interpretability, efficiency, and adaptability to schema changes. By conducting
extensive experiments on our newly curated datasets, we pinpoint the
instability issues within ChatGPT and highlight the superior performance of
ZSP. ZSP achieves an impressive 40% improvement in F1 score for fine-grained
Rootcode classification. ZSP demonstrates competitive performance compared to
supervised BERT models, positioning it as a valuable tool for event record
validation and ontology development. Our work underscores the potential of
leveraging transfer learning and existing expertise to enhance the efficiency
and scalability of research in the field.

最近的事件编码监督模型在超过模式匹配方法的性能。然而，它们仅依赖新注释，忽视了专家数据库中的丰富知识，限制了其在细粒度分类中的适用性。为了解决这些限制，我们探索了从已建立的注释代码库中获取知识的零样本方法，用于政治事件本体关系分类。我们的研究涵盖了 ChatGPT 和一种新的基于自然语言推断（NLI）的方法，名为 ZSP。ZSP 采用了一种树查询框架，将任务分解为上下文、语态和类别消歧的层次。该框架提高了可解释性、效率和适应架构变化。通过对我们新精心准备的数据集进行广泛实验，我们确定了 ChatGPT 内的不稳定性问题，并突出了 ZSP 优越的性能。ZSP 在细粒度 Rootcode 分类的 F1 得分上取得了 40% 的显著改进。与监督的 BERT 模型相比，ZSP 表现出了竞争性能，使其成为事件记录验证和本体发展的有价值工具。我们的工作强调了利用迁移学习和现有专业知识提高领域研究的效率和可扩展性的潜力。

通过代码簿知识、NLI 和 ChatGPT 综合合成政治零射关系分类

Synthesizing Political Zero-Shot Relation Classification via Codebook  Knowledge, NLI, and ChatGPT

For monitoring crises, political events are extracted from the news. The
large amount of unstructured full-text event descriptions makes a case-by-case
analysis unmanageable, particularly for low-resource humanitarian aid
organizations. This creates a demand to classify events into event types, a
task referred to as event coding. Typically, domain experts craft an event type
ontology, annotators label a large dataset and technical experts develop a
supervised coding system. In this work, we propose PR-ENT, a new event coding
approach that is more flexible and resource-efficient, while maintaining
competitive accuracy: first, we extend an event description such as "Military
injured two civilians'' by a template, e.g. "People were [Z]" and prompt a
pre-trained (cloze) language model to fill the slot Z. Second, we select answer
candidates Z* = {"injured'', "hurt"...} by treating the event description as
premise and the filled templates as hypothesis in a textual entailment task.
This allows domain experts to draft the codebook directly as labeled prompts
and interpretable answer candidates. This human-in-the-loop process is guided
by our interactive codebook design tool. We evaluate PR-ENT in several
robustness checks: perturbing the event description and prompt template,
restricting the vocabulary and removing contextual information.

本文提出了一种名为 PR-ENT 的事件编码新方法，通过引入预训练语言模型对事件描述进行扩展，并将事件描述视为前提，填充的模板作为假设，从中提取可解释的答案，从而可直接起草编码手册，并通过交互式编码手册设计工具进行人机交互，从而更灵活、高效地对事件进行编码。