Pretrained contextual and non-contextual subword embeddings have become available in over 250 languages, allowing massively multilingual NLP. However, while there is no dearth of pretrained embeddings, the distinct lack of systematic evaluations makes it difficult for practitioners to choose between them. In this work, we conduct an extensive evaluation comparing non-contextual subword embeddings, namely FastText and BPEmb, and a contextual representation method, namely BERT, on multilingual named entity recognition and part-of-speech tagging. We find that overall, a combination of BERT, BPEmb, and character representations works best across languages and tasks. A more detailed analysis reveals different strengths and weaknesses: Multilingual BERT performs well in medium- to high-resource languages, but is outperformed by non-contextual subword embeddings in a low-resource setting.

本文对多语种命名实体识别和词性标注任务中预训练的 BERT、BPEmb 和 FastText 非上下文子词嵌入进行系统性评估，发现对于跨语言和任务而言，BERT、BPEmb 和字符表示的组合效果最佳，但在资源匮乏的情况下，非上下文子词嵌入的表现更加优秀。

使用上下文和非上下文表示的序列标注：一项多语言评估