We propose procedures for evaluating and strengthening contextual embedding
alignment and show that they are useful in analyzing and improving multilingual
BERT. In particular, after our proposed alignment procedure, BERT exhibits
significantly improved zero-shot performance on XNLI compared to the base
model, remarkably matching pseudo-fully-supervised translate-train models for
Bulgarian and Greek. Further, to measure the degree of alignment, we introduce
a contextual version of word retrieval and show that it correlates well with
downstream zero-shot transfer. Using this word retrieval task, we also analyze
BERT and find that it exhibits systematic deficiencies, e.g. worse alignment
for open-class parts-of-speech and word pairs written in different scripts,
that are corrected by the alignment procedure. These results support contextual
alignment as a useful concept for understanding large multilingual pre-trained
models.

提出了一种测量和加强上下文嵌入对齐的程序，证明其在分析和改进多语言 BERT 方面非常有用。在我们的对齐程序之后，BERT 在零 - shot 任务中表现显著提高。使用上下文版本的 Word Retrieval 任务，我们发现 BERT 存在系统性缺陷，在我们的对齐程序下得到了纠正。这些结果支持上下文对齐作为理解大型多语言预训练模型的一个有用概念。