Multilingual evaluation benchmarks usually contain limited high-resource
languages and do not test models for specific linguistic capabilities.
CheckList is a template-based evaluation approach that tests models for
specific capabilities. The CheckList template creation process requires native
speakers, posing a challenge in scaling to hundreds of languages. In this work,
we explore multiple approaches to generate Multilingual CheckLists. We device
an algorithm - Template Extraction Algorithm (TEA) for automatically extracting
target language CheckList templates from machine translated instances of a
source language templates. We compare the TEA CheckLists with CheckLists
created with different levels of human intervention. We further introduce
metrics along the dimensions of cost, diversity, utility, and correctness to
compare the CheckLists. We thoroughly analyze different approaches to creating
CheckLists in Hindi. Furthermore, we experiment with 9 more different
languages. We find that TEA followed by human verification is ideal for scaling
Checklist-based evaluation to multiple languages while TEA gives a good
estimates of model performance.

本文介绍了一种使用机器翻译自动提取目标语言的 CheckList 模板的算法，以及将其与不同水平的人工干预下创建的 CheckList 进行比较的度量方法，并探讨对印地语和其他九种语言进行 CheckList 创作的不同方法，最终得出 TEA 加上人工干预的方法最适合将基于 Checklist 的评估扩展到多种语言，并且通过 TEA 可以很好地估算模型的性能。