Multilingual BERT (mBERT) has shown reasonable capability for zero-shot
cross-lingual transfer when fine-tuned on downstream tasks. Since mBERT is not
pre-trained with explicit cross-lingual supervision, transfer performance can
further be improved by aligning mBERT with cross-lingual signal. Prior work
proposes several approaches to align contextualised embeddings. In this paper
we analyse how different forms of cross-lingual supervision and various
alignment methods influence the transfer capability of mBERT in zero-shot
setting. Specifically, we compare parallel corpora vs. dictionary-based
supervision and rotational vs. fine-tuning based alignment methods. We evaluate
the performance of different alignment methodologies across eight languages on
two tasks: Name Entity Recognition and Semantic Slot Filling. In addition, we
propose a novel normalisation method which consistently improves the
performance of rotation-based alignment including a notable 3% F1 improvement
for distant and typologically dissimilar languages. Importantly we identify the
biases of the alignment methods to the type of task and proximity to the
transfer language. We also find that supervision from parallel corpus is
generally superior to dictionary alignments.

本文分析了不同形式的跨语言监督和各种对齐方法对 Multilingual BERT 在零 - shot 设置下的转移能力的影响，比较了平行语料库 vs. 字典导向的监督和旋转 vs. 微调对齐方法，针对两个任务（名称实体识别和语义槽填充），我们评估了不同对齐方法在八种语言上的性能，并提出了一种新的规范化方法，该方法不断改进基于旋转的对齐性能，特别是远距离和在语言类型上不同的语言。此外，我们还发现对于任务和转移语言的接近程度，不同的对齐方法存在偏差，平行语料库的监督力度通常优于字典对齐。