Parameter efficient finetuning has emerged as a viable solution for improving
the performance of Large Language Models without requiring massive resources
and compute. Prior work on multilingual evaluation has shown that there is a
large gap between the performance of LLMs on English and other languages.
Further, there is also a large gap between the performance of smaller
open-source models and larger LLMs. Finetuning can be an effective way to
bridge this gap and make language models more equitable. In this work, we
finetune the LLaMA-7B and Mistral-7B models on synthetic multilingual
instruction tuning data to determine its effect on model performance on five
downstream tasks covering twenty three languages in all. Additionally, we
experiment with various parameters, such as rank for low-rank adaptation and
values of quantisation to determine their effects on downstream performance and
find that higher rank and higher quantisation values benefit low-resource
languages. We find that parameter efficient finetuning of smaller open source
models sometimes bridges the gap between the performance of these models and
the larger ones, however, English performance can take a hit. We also find that
finetuning sometimes improves performance on low-resource languages, while
degrading performance on high-resource languages.

通过参数高效微调能够提高大语言模型的性能，而无需大量的资源和计算。对多语言评估的先前研究表明，英语和其他语言在性能上存在很大差距。此外，开源模型和较大规模的语言模型之间也存在差距。微调是弥合差距、使语言模型更加公平的有效方法。本研究通过对合成多语言指示微调数据上进行 LLaMA-7B 和 Mistral-7B 模型的微调，以评估其对模型在涵盖 23 种语言的五个下游任务上的性能影响。此外，我们还在低秩适应的等级和量化值上进行了实验，以确定它们对下游性能的影响并发现较高的等级和量化值有益于低资源语言。我们发现通过参数高效微调较小的开源模型有时可以弥合这些模型和较大模型性能之间的差距，但对英语性能有所降低。我们还发现微调有时可以提高低资源语言的性能，但在高资源语言上性能可能会下降。

MAPLE: 多语言大语言模型参数高效微调的评估

MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of  Large Language Models

Large language models (LLMs) can use in-context demonstrations to improve
performance on zero-shot tasks. However, selecting the best in-context examples
is challenging because model performance can vary widely depending on the
selected examples. We present a cross-entropy difference (CED) method for
selecting in-context demonstrations. Our method is based on the observation
that the effectiveness of in-context demonstrations negatively correlates with
the perplexity of the test example by a language model that was finetuned on
that demonstration. We utilize parameter efficient finetuning to train small
models on training data that are used for computing the cross-entropy
difference between a test example and every candidate in-context demonstration.
This metric is used to rank and select in-context demonstrations independently
for each test input. We evaluate our method on a mix-domain dataset that
combines 8 benchmarks, representing 4 text generation tasks, showing that CED
for in-context demonstration selection can improve performance for a variety of
LLMs.

本研究提出了一个基于交叉熵差异的方法，用于选择上下文演示文稿以提高语言模型的性能。该方法基于该观察结果：在特定演示文稿上进行微调的语言模型在测试样例上的困惑度与上下文演示的有效性呈负相关。研究者评估了该方法在混合域数据集上的表现，并表明该方法可提高各种大型语言模型的性能。