Despite their success, modern language models are fragile. Even small changes
in their training pipeline can lead to unexpected results. We study this
phenomenon by examining the robustness of ALBERT (arXiv:1909.11942) in
combination with Stochastic Weight Averaging (SWA) (arXiv:1803.05407) -- a
cheap way of ensembling -- on a sentiment analysis task (SST-2). In particular,
we analyze SWA's stability via CheckList criteria (arXiv:2005.04118), examining
the agreement on errors made by models differing only in their random seed. We
hypothesize that SWA is more stable because it ensembles model snapshots taken
along the gradient descent trajectory. We quantify stability by comparing the
models' mistakes with Fleiss' Kappa (Fleiss, 1971) and overlap ratio scores. We
find that SWA reduces error rates in general; yet the models still suffer from
their own distinct biases (according to CheckList).

本研究研究了模型的鲁棒性，特别是在情绪分析任务上的鲁棒性，通过研究 ALBERT 与 SWA 的结合来分析 SWA 的稳定性，通过 CheckList 准则量化稳定性，发现 SWA 不仅可以降低错误率，还可以集成模型贯穿梯度下降轨迹的快照来提高稳定性。