Human label variation (HLV) is a valuable source of information that arises
when multiple human annotators provide different labels for valid reasons. In
Natural Language Inference (NLI) earlier approaches to capturing HLV involve
either collecting annotations from many crowd workers to represent human
judgment distribution (HJD) or use expert linguists to provide detailed
explanations for their chosen labels. While the former method provides denser
HJD information, obtaining it is resource-intensive. In contrast, the latter
offers richer textual information but it is challenging to scale up to many
human judges. Besides, large language models (LLMs) are increasingly used as
evaluators (``LLM judges'') but with mixed results, and few works aim to study
HJDs. This study proposes to exploit LLMs to approximate HJDs using a small
number of expert labels and explanations. Our experiments show that a few
explanations significantly improve LLMs' ability to approximate HJDs with and
without explicit labels, thereby providing a solution to scale up annotations
for HJD. However, fine-tuning smaller soft-label aware models with the
LLM-generated model judgment distributions (MJDs) presents partially
inconsistent results: while similar in distance, their resulting fine-tuned
models and visualized distributions differ substantially. We show the
importance of complementing instance-level distance measures with a
global-level shape metric and visualization to more effectively evaluate MJDs
against human judgment distributions.

利用少数专家标签和解释，借助大规模语言模型 (LLM) 近似人类判断分布 (HJD), 从而提供了一种对 HJD 进行标注扩展的解决方案。然而，利用 LLM 生成的判断分布 (MJD) 进行微调时，结果存在部分不一致性，需要同时使用全局级的形状度量和可视化来更有效地评估 MJD 和 HJD 的一致性。