BriefGPT.xyz
Mar, 2022
k-一致性系数：用于人类标注数据的正确可靠性单位
k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations
HTML
PDF
Ka Wong, Praveen Paritosh
TL;DR
本文讨论了聚合策略在应对不可靠数据上的应用，并提出了k-评分者可靠性来探讨以聚合评分作为数据可靠性的正确单位；作者进行了WordSim-353基准测试并提出了计算k-评分者可靠性的方法，强调了在汇报可靠性时应同时报告k-评分者可靠性和评分者间可靠性。
Abstract
Since the inception of
crowdsourcing
, aggregation has been a common strategy for dealing with unreliable data. Aggregate ratings are more reliable than individual ones. However, many
natural language processing
(
→