In evidence-based medicine, relevance of medical literature is determined by
predefined relevance conditions. The conditions are defined based on PICO
elements, namely, Patient, Intervention, Comparator, and Outcome. Hence, PICO
annotations in medical literature are essential for automatic relevant document
filtering. However, defining boundaries of text spans for PICO elements is not
straightforward. In this paper, we study the agreement of PICO annotations made
by multiple human annotators, including both experts and non-experts.
Agreements are estimated by a standard span agreement (i.e., matching both
labels and boundaries of text spans), and two types of relaxed span agreement
(i.e., matching labels without guaranteeing matching boundaries of spans).
Based on the analysis, we report two observations: (i) Boundaries of PICO span
annotations by individual human annotators are very diverse. (ii) Despite the
disagreement in span boundaries, general areas of the span annotations are
broadly agreed by annotators. Our results suggest that applying a standard
agreement alone may undermine the agreement of PICO spans, and adopting both a
standard and a relaxed agreements is more suitable for PICO span evaluation.

本文研究了由多个人工标注员（包括专家和非专家）进行的 PICO 注释的一致性，并发现个体标注员的注释边界非常多样化，认为仅使用标准协议可能会削弱 PICO 跨度的一致性，同时采用标准和宽松协议更适用于 PICO 跨度评估。

PICO 跨度标注中的一致性研究

A Study on Agreement in PICO Span Annotations

Filtering relevant documents with respect to entities is an essential task in
the context of knowledge base construction and maintenance. It entails
processing a time-ordered stream of documents that might be relevant to an
entity in order to select only those that contain vital information.
State-of-the-art approaches to document filtering for popular entities are
entity-dependent: they rely on and are also trained on the specifics of
differentiating features for each specific entity. Moreover, these approaches
tend to use so-called extrinsic information such as Wikipedia page views and
related entities which is typically only available only for popular head
entities. Entity-dependent approaches based on such signals are therefore
ill-suited as filtering methods for long-tail entities. In this paper we
propose a document filtering method for long-tail entities that is
entity-independent and thus also generalizes to unseen or rarely seen entities.
It is based on intrinsic features, i.e., features that are derived from the
documents in which the entities are mentioned. We propose a set of features
that capture informativeness, entity-saliency, and timeliness. In particular,
we introduce features based on entity aspect similarities, relation patterns,
and temporal expressions and combine these with standard features for document
filtering. Experiments following the TREC KBA 2014 setup on a publicly
available dataset show that our model is able to improve the filtering
performance for long-tail entities over several baselines. Results of applying
the model to unseen entities are promising, indicating that the model is able
to learn the general characteristics of a vital document. The overall
performance across all entities---i.e., not just long-tail entities---improves
upon the state-of-the-art without depending on any entity-specific training
data.

该论文提出了一种基于文献内在特征的长尾实体文档过滤方法，通过使用与实体相关的方面相似性、关系模式和时间表达式等标准特征，结合信息量、实体显著性和时效性等一些特征，能够在不依赖任何实体特定的训练数据的情况下提高过滤性能。