BriefGPT.xyz
Jun, 2021
交叉复制可靠性--解释评定者间可靠性的经验方法
Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability
HTML
PDF
Ka Wong, Praveen Paritosh, Lora Aroyo
TL;DR
提出了一种称为xRR框架的方法，通过在复制实验中将IRR与基准测量进行基准测试，其中包括基于Cohen的kappa的新型交叉复制可靠性（xRR）测量，我们将其用于衡量众包数据集的质量。对4百万人类对面部表情的判断进行了分析。
Abstract
We present a new approach to interpreting
irr
that is empirical and contextualized. It is based upon benchmarking
irr
against baseline measures in a
→