Larger language models have higher accuracy on average, but are they better
on every single instance (datapoint)? Some work suggests larger models have
higher out-of-distribution robustness, while other work suggests they have
lower accuracy on rare subgroups. To understand these differences, we
investigate these models at the level of individual instances. However, one
major challenge is that individual predictions are highly sensitive to noise in
the randomness in training. We develop statistically rigorous methods to
address this, and after accounting for pretraining and finetuning noise, we
find that our BERT-Large is worse than BERT-Mini on at least 1-4% of instances
across MNLI, SST-2, and QQP, compared to the overall accuracy improvement of
2-10%. We also find that finetuning noise increases with model size and that
instance-level accuracy has momentum: improvement from BERT-Mini to BERT-Medium
correlates with improvement from BERT-Medium to BERT-Large. Our findings
suggest that instance-level predictions provide a rich source of information;
we therefore, recommend that researchers supplement model weights with model
predictions.

比较不同规模的语言模型在不同实例上的表现，发现 BERT-Large 在一定比例上比 BERT-Mini 表现差。建议研究人员结合模型预测来分析模型表现。