We study estimating inherent human disagreement (annotation label
distribution) in natural language inference task. Post-hoc smoothing of the
predicted label distribution to match the expected label entropy is very
effective. Such simple manipulation can reduce KL divergence by almost half,
yet will not improve majority label prediction accuracy or learn label
distributions. To this end, we introduce a small amount of examples with
multiple references into training. We depart from the standard practice of
collecting a single reference per each training example, and find that
collecting multiple references can achieve better accuracy under the fixed
annotation budget. Lastly, we provide rich analyses comparing these two methods
for improving label distribution estimation.

研究了自然语言推理任务中固有人类分歧 (注释标签分布) 的估计。通过后处理平滑预测的标签分布以匹配期望的标签熵取得了很好的效果。同时，通过引入有多个参考的少量样例进行训练，相较于传统做法每个训练样本只采集一个参考，我们发现这种多参考的方法可以在固定注释预算下实现更好的精度。最后，提供了对比这两种方法的丰富分析，用于改进标签分布估计。