Although recent Massively Multilingual Language Models (MMLMs) like mBERT and
XLMR support around 100 languages, most existing multilingual NLP benchmarks
provide evaluation data in only a handful of these languages with little
linguistic diversity. We argue that this makes the existing practices in
multilingual evaluation unreliable and does not provide a full picture of the
performance of MMLMs across the linguistic landscape. We propose that the
recent work done in Performance Prediction for NLP tasks can serve as a
potential solution in fixing benchmarking in Multilingual NLP by utilizing
features related to data and language typology to estimate the performance of
an MMLM on different languages. We compare performance prediction with
translating test data with a case study on four different multilingual
datasets, and observe that these methods can provide reliable estimates of the
performance that are often on-par with the translation based approaches,
without the need for any additional translation as well as evaluation costs.

本文提出了一种利用语言数据和语言类型学特征来预测跨语种语言模型性能的方法，以此取代传统基于翻译的方法评估系统，该方法表现良好并且能够可靠地估计模型在不同语言上的表现。