If the same neural architecture is trained multiple times on the same dataset, will it make similar linguistic generalizations across runs? To study this question, we fine-tuned 100 instances of BERT on the Multi-genre Natural Language Inference (MNLI) dataset and evaluated them on the HANS dataset, which measures syntactic generalization in natural language inference. On the MNLI development set, the behavior of all instances was remarkably consistent, with accuracy ranging between 83.6% and 84.8%. In stark contrast, the same models varied widely in their generalization performance. For example, on the simple case of subject-object swap (e.g., knowing that "the doctor visited the lawyer" does not entail "the lawyer visited the doctor"), accuracy ranged from 0.00% to 66.2%. Such variation likely arises from the presence of many local minima that are equally attractive to a low-bias learner such as a neural network; decreasing the variability may therefore require models with stronger inductive biases.

本研究研究了同一神经网络架构在同一数据集上多次训练后是否在不同运行中进行相似的语言概括，通过在Multi-genre自然语言推理（MNLI）数据集上微调100个BERT实例并在评估自然语言推理中的句法概括的HANS数据集上对它们进行了评估，这类变异很可能是由于在局部极小值处的神经网络中出现的损失函数梯度等的不同选择而存在的，而减少变异可能需要具有更强归纳偏差的模型。

相似性能下的BERT模型泛化能力差异大