We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. Using this word retrieval task, we also analyze BERT and find that it exhibits systematic deficiencies, e.g. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure. These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models.

提出了一种测量和加强上下文嵌入对齐的程序，证明其在分析和改进多语言BERT方面非常有用。在我们的对齐程序之后，BERT在零-shot任务中表现显著提高。使用上下文版本的Word Retrieval任务，我们发现BERT存在系统性缺陷，在我们的对齐程序下得到了纠正。这些结果支持上下文对齐作为理解大型多语言预训练模型的一个有用概念。

多语种上下文词表示的对齐