Manual data annotation is an important NLP task but one that takes
considerable amount of resources and effort. In spite of the costs, labeling
and categorizing entities is essential for NLP tasks such as semantic
evaluation. Even though annotation can be done by non-experts in most cases,
due to the fact that this requires human labor, the process is costly. Another
major challenge encountered in data annotation is maintaining the annotation
consistency. Annotation efforts are typically carried out by teams of multiple
annotators. The annotations need to maintain the consistency in relation to
both the domain truth and annotation format while reducing human errors.
Annotating a specialized domain that deviates significantly from the general
domain, such as fantasy literature, will see a lot of human error and annotator
disagreement. So it is vital that proper guidelines and error reduction
mechanisms are enforced. One such way to enforce these constraints is using a
specialized application. Such an app can ensure that the notations are
consistent, and the labels can be pre-defined or restricted reducing the room
for errors. In this paper, we present SHADE, an annotation software that can be
used to annotate entities in the high fantasy literature domain. Specifically
in Dungeons and Dragons lore extracted from the Forgotten Realms Fandom Wiki.

手动数据标注是一项重要的自然语言处理任务，但需要大量资源和工作。在数据标注中，保持标注的一致性是一个主要挑战，因此需要一种专门的应用程序来确保标注的一贯性，并减少人为错误。本文介绍了一种名为 SHADE 的专门用于高奇幻文学领域中的实体标注的软件。

SHADE: 领域特定实体的语义上位词注释工具 – DnD 领域应用案例

SHADE: Semantic Hypernym Annotator for Domain-specific Entities -- DnD  Domain Use Case

MultiWOZ is one of the most popular multi-domain task-oriented dialog
datasets, containing 10K+ annotated dialogs covering eight domains. It has been
widely accepted as a benchmark for various dialog tasks, e.g., dialog state
tracking (DST), natural language generation (NLG), and end-to-end (E2E) dialog
modeling. In this work, we identify an overlooked issue with dialog state
annotation inconsistencies in the dataset, where a slot type is tagged
inconsistently across similar dialogs leading to confusion for DST modeling. We
propose an automated correction for this issue, which is present in a whopping
70% of the dialogs. Additionally, we notice that there is significant entity
bias in the dataset (e.g., "cambridge" appears in 50% of the destination cities
in the train domain). The entity bias can potentially lead to named entity
memorization in generative models, which may go unnoticed as the test set
suffers from a similar entity bias as well. We release a new test set with all
entities replaced with unseen entities. Finally, we benchmark joint goal
accuracy (JGA) of the state-of-the-art DST baselines on these modified versions
of the data. Our experiments show that the annotation inconsistency corrections
lead to 7-10% improvement in JGA. On the other hand, we observe a 29% drop in
JGA when models are evaluated on the new test set with unseen entities.

本研究针对 MultiWOZ 数据集中 Dialog State Tracking (DST) 这一任务的注释一致性问题进行了研究，并提出了一种自动的纠正方法，通过该方法可以在数据集上进行 7-10％的 JGA 增强。此外，还检测到数据集中的实体偏差问题，并在新的测试集上测试出实体未知时，JGA 可能下降高达 29％。