Multilingual pre-trained Transformers, such as mBERT (Devlin et al., 2019)
and XLM-RoBERTa (Conneau et al., 2020a), have been shown to enable the
effective cross-lingual zero-shot transfer. However, their performance on
Arabic information extraction (IE) tasks is not very well studied. In this
paper, we pre-train a customized bilingual BERT, dubbed GigaBERT, that is
designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer
learning. We study GigaBERT's effectiveness on zero-short transfer across four
IE tasks: named entity recognition, part-of-speech tagging, argument role
labeling, and relation extraction. Our best model significantly outperforms
mBERT, XLM-RoBERTa, and AraBERT (Antoun et al., 2020) in both the supervised
and zero-shot transfer settings. We have made our pre-trained models publicly
available at this https URL

本文研究了针对阿拉伯自然语言处理和英语到阿拉伯语的零样本迁移学习设计的定制双语 BERT 模型 - GigaBERT，重点研究了它在四个信息抽取任务中的零样本迁移效果，实验证明该模型在监督和零样本迁移设置下都显著优于 mBERT、XLM-RoBERTa 和 AraBERT。