The rise of large language models (LLMs) has brought a critical need for
high-quality human-labeled data, particularly for processes like human feedback
and evaluation. A common practice is to label data via consensus annotation
over the judgments of multiple crowdworkers. However, different annotators may
have different interpretations of labeling schemes unless given extensive
training, and for subjective NLP tasks, even trained expert annotators can
diverge heavily. We show that these nuances can be captured by high quality
natural language explanations, and propose a method to rescale ordinal
annotation in the presence of disagreement using LLMs. Specifically, we feed
Likert ratings and corresponding natural language explanations into an LLM and
prompt it to produce a numeric score. This score should reflect the underlying
assessment of the example by the annotator. The presence of explanations allows
the LLM to homogenize ratings across annotators in spite of scale usage
differences. We explore our technique in the context of a document-grounded
question answering task on which large language models achieve near-human
performance. Among questions where annotators identify incompleteness in the
answers, our rescaling improves correlation between nearly all annotator pairs,
improving pairwise correlation on these examples by an average of 0.2 Kendall's
tau.

本文提出了一种利用自然语言解释对大型语言模型进行训练的方法，通过收集一致性注释来标记数据，以及利用该方法对大型语言模型进行训练，从而提高标注数据的质量。