Borrowing ideas from {\em Production functions} in micro-economics, in this
paper we introduce a framework to systematically evaluate the performance and
cost trade-offs between machine-translated and manually-created labelled data
for task-specific fine-tuning of massively multilingual language models. We
illustrate the effectiveness of our framework through a case-study on the
TyDIQA-GoldP dataset. One of the interesting conclusions of the study is that
if the cost of machine translation is greater than zero, the optimal
performance at least cost is always achieved with at least some or only
manually-created data. To our knowledge, this is the first attempt towards
extending the concept of production functions to study data collection
strategies for training multilingual models, and can serve as a valuable tool
for other similar cost vs data trade-offs in NLP.

本研究引用微观经济学中的生产函数理论，在机器翻译和手动标注数据的建模中提出一种系统评估模型性能和成本折衷的框架，并通过 TyDIQA-GoldP 数据集的案例研究证明了该框架的有效性和结论为手动标注数据的性价比优于或至少等于机器翻译。此外，这也是第一次尝试将生产函数理论扩展到 NLP 中的数据收集策略研究，并能为类似的成本 vs 数据折衷问题提供有价值的工具。

关于多语言少样本学习的经济学：机器翻译和手动数据的成本 - 性能权衡建模

On the Economics of Multilingual Few-shot Learning: Modeling the Cost-Performance Trade-offs of Machine Translated and Manual Data

Although recent Massively Multilingual Language Models (MMLMs) like mBERT and
XLMR support around 100 languages, most existing multilingual NLP benchmarks
provide evaluation data in only a handful of these languages with little
linguistic diversity. We argue that this makes the existing practices in
multilingual evaluation unreliable and does not provide a full picture of the
performance of MMLMs across the linguistic landscape. We propose that the
recent work done in Performance Prediction for NLP tasks can serve as a
potential solution in fixing benchmarking in Multilingual NLP by utilizing
features related to data and language typology to estimate the performance of
an MMLM on different languages. We compare performance prediction with
translating test data with a case study on four different multilingual
datasets, and observe that these methods can provide reliable estimates of the
performance that are often on-par with the translation based approaches,
without the need for any additional translation as well as evaluation costs.

本文提出了一种利用语言数据和语言类型学特征来预测跨语种语言模型性能的方法，以此取代传统基于翻译的方法评估系统，该方法表现良好并且能够可靠地估计模型在不同语言上的表现。