Multilingual contextual embeddings have demonstrated state-of-the-art
performance in zero-shot cross-lingual transfer learning, where multilingual
BERT is fine-tuned on one source language and evaluated on a different target
language. However, published results for mBERT zero-shot accuracy vary as much
as 17 points on the MLDoc classification task across four papers. We show that
the standard practice of using English dev accuracy for model selection in the
zero-shot setting makes it difficult to obtain reproducible results on the
MLDoc and XNLI tasks. English dev accuracy is often uncorrelated (or even
anti-correlated) with target language accuracy, and zero-shot performance
varies greatly at different points in the same fine-tuning run and between
different fine-tuning runs. These reproducibility issues are also present for
other tasks with different pre-trained embeddings (e.g., MLQA with XLM-R). We
recommend providing oracle scores alongside zero-shot results: still fine-tune
using English data, but choose a checkpoint with the target dev set. Reporting
this upper bound makes results more consistent by avoiding arbitrarily bad
checkpoints.

本文研究了多语言语境嵌入在零样本跨语言迁移学习上的表现，并发现使用英语进行零样本设置时很难获得可重复的结果和一致的检查点，建议在零样本结果旁边提供 oracle 分数，通过避免任意差的检查点使结果更一致。