Natural language understanding (NLU) tasks face a non-trivial amount of
ambiguous samples where veracity of their labels is debatable among annotators.
NLU models should thus account for such ambiguity, but they approximate the
human opinion distributions quite poorly and tend to produce over-confident
predictions. To address this problem, we must consider how to exactly capture
the degree of relationship between each sample and its candidate classes. In
this work, we propose a novel method with deep model compression and show how
such relationship can be accounted for. We see that more reasonably represented
relationships can be discovered in the lower layers and that validation
accuracies are converging at these layers, which naturally leads to layer
pruning. We also see that distilling the relationship knowledge from a lower
layer helps models produce better distribution. Experimental results
demonstrate that our method makes substantial improvement on quantifying
ambiguity without gold distribution labels. As positive side-effects, our
method is found to reduce the model size significantly and improve latency,
both attractive aspects of NLU products.

本研究提出了一种基于深度模型压缩的新方法，解决了自然语言理解中标记不确定性的问题，并发现了在低层中更合理表示的关系，这有助于减小模型的规模并改善可用性。