We present a new corpus with coreference annotation, Russian Coreference
Corpus (RuCoCo). The goal of RuCoCo is to obtain a large number of annotated
texts while maintaining high inter-annotator agreement. RuCoCo contains news
texts in Russian, part of which were annotated from scratch, and for the rest
the machine-generated annotations were refined by human annotators. The size of
our corpus is one million words and around 150,000 mentions. We make the corpus
publicly available.

我们提出了一个带有指代消解注释的新语料库，俄语指代消解语料库 (RuCoCo)。RuCoCo 旨在在保持高注释者一致性的同时获取大量标注的文本。我们的语料库包含了俄语的新闻文本，其中部分是从头开始注释的，而其余部分则是由机器生成的注释被人工注释精细化的。我们的语料库包含了一百万个单词和约 15 万个提及，现已公开发布。

RuCoCo：一个新的俄语语料库，带有指代标注

RuCoCo: a new Russian corpus with coreference annotation

In evidence-based medicine, relevance of medical literature is determined by
predefined relevance conditions. The conditions are defined based on PICO
elements, namely, Patient, Intervention, Comparator, and Outcome. Hence, PICO
annotations in medical literature are essential for automatic relevant document
filtering. However, defining boundaries of text spans for PICO elements is not
straightforward. In this paper, we study the agreement of PICO annotations made
by multiple human annotators, including both experts and non-experts.
Agreements are estimated by a standard span agreement (i.e., matching both
labels and boundaries of text spans), and two types of relaxed span agreement
(i.e., matching labels without guaranteeing matching boundaries of spans).
Based on the analysis, we report two observations: (i) Boundaries of PICO span
annotations by individual human annotators are very diverse. (ii) Despite the
disagreement in span boundaries, general areas of the span annotations are
broadly agreed by annotators. Our results suggest that applying a standard
agreement alone may undermine the agreement of PICO spans, and adopting both a
standard and a relaxed agreements is more suitable for PICO span evaluation.

本文研究了由多个人工标注员（包括专家和非专家）进行的 PICO 注释的一致性，并发现个体标注员的注释边界非常多样化，认为仅使用标准协议可能会削弱 PICO 跨度的一致性，同时采用标准和宽松协议更适用于 PICO 跨度评估。