Deep Neural Networks (DNNs) are becoming integral components of real world
services relied upon by millions of users. Unfortunately, architects of these
systems can find it difficult to ensure reliable performance as irrelevant
details like random initialization can unexpectedly change the outputs of a
trained system with potentially disastrous consequences. We formulate the model
stability problem by studying how the predictions of a model change, even when
it is retrained on the same data, as a consequence of stochasticity in the
training process. For Natural Language Understanding (NLU) tasks, we find
instability in predictions for a significant fraction of queries. We formulate
principled metrics, like per-sample ``label entropy'' across training runs or
within a single training run, to quantify this phenomenon. Intriguingly, we
find that unstable predictions do not appear at random, but rather appear to be
clustered in data-specific ways. We study data-agnostic regularization methods
to improve stability and propose new data-centric methods that exploit our
local stability estimates. We find that our localized data-specific mitigation
strategy dramatically outperforms data-agnostic methods, and comes within 90%
of the gold standard, achieved by ensembling, at a fraction of the
computational cost

通过研究模型在训练过程中的随机性，我们发现在自然语言理解任务中，模型的预测结果的不稳定性会对一部分查询产生影响，提出了一种衡量稳定性现象的方法 —— 标签熵，我们研究了数据不可知的正则化方法和新的数据中心方法，发现它已在计算成本的分数中取得了 90% 的性能优势以及金标准的表现。