Clinical studies often require understanding elements of a patient's
narrative that exist only in free text clinical notes. To transform notes into
structured data for downstream use, these elements are commonly extracted and
normalized to medical vocabularies. In this work, we audit the performance of
and indicate areas of improvement for state-of-the-art systems. We find that
high task accuracies for clinical entity normalization systems on the 2019 n2c2
Shared Task are misleading, and underlying performance is still brittle.
Normalization accuracy is high for common concepts (95.3%), but much lower for
concepts unseen in training data (69.3%). We demonstrate that current
approaches are hindered in part by inconsistencies in medical vocabularies,
limitations of existing labeling schemas, and narrow evaluation techniques. We
reformulate the annotation framework for clinical entity extraction to factor
in these issues to allow for robust end-to-end system benchmarking. We evaluate
concordance of annotations from our new framework between two annotators and
achieve a Jaccard similarity of 0.73 for entity recognition and an agreement of
0.83 for entity normalization. We propose a path forward to address the
demonstrated need for the creation of a reference standard to spur method
development in entity recognition and normalization.

本研究运用新的注释框架拟定临床实体抽取的评估标准，并发现现有的医学词汇一致性、数据标签限制、评估技术瓶颈等问题仍需要解决，因此提出了解决方案来推动临床实体识别和标准化的发展。