Zero-shot cross-lingual generation implies finetuning of the multilingual
pretrained language model on a generation task in one language and then using
it to make predictions for this task in other languages. Previous works notice
a frequent problem of generation in a wrong language and propose approaches to
address it, usually using mT5 as a backbone model. In this work we compare
various approaches proposed from the literature in unified settings, also
including alternative backbone models, namely mBART and NLLB-200. We first
underline the importance of tuning learning rate used for finetuning, which
helps to substantially alleviate the problem of generation in the wrong
language. Then, we show that with careful learning rate tuning, the simple full
finetuning of the model acts as a very strong baseline and alternative
approaches bring only marginal improvements. Finally, we find that mBART
performs similarly to mT5 of the same size, and NLLB-200 can be competitive in
some cases. Our final models reach the performance of the approach based on
data translation which is usually considered as an upper baseline for zero-shot
cross-lingual generation.

零 - shot 跨语言生成的前提是在一个语言上对多语言预训练语言模型进行微调，并将其用于其他语言的生成任务中。先前的研究注意到生成错误语言的频繁问题，并提出了解决方法，通常使用 mT5 作为基础模型。本研究在统一设置下比较了文献中提出的各种方法，还包括 mBART 和 NLLB-200 作为备用基础模型。我们首先强调了微调中使用的学习率调整的重要性，这有助于大幅减轻生成错误语言的问题。然后，我们展示了通过仔细调整学习率，简单的模型全面微调作为一个非常强大的基准，备选方法带来的只是微小的改进。最后，我们发现 mBART 的性能与相同大小的 mT5 相似，并且在某些情况下，NLLB-200 可以有竞争力。我们的最终模型达到了基于数据翻译的方法的性能水平，这通常被视为零 - shot 跨语言生成的上限基准。

生成任务中有效的零射击跨语言知识转移的关键要素

Key ingredients for effective zero-shot cross-lingual knowledge transfer  in generative tasks

Zero-shot cross-lingual generation assumes finetuning the multilingual
pretrained language model (mPLM) on a generation task in one language and then
using it to make predictions for this task in other languages. Previous works
notice a frequent problem of generation in a wrong language and propose
approaches to address it, usually using mT5 as a backbone model. In this work,
we test alternative mPLMs, such as mBART and NLLB, considering full finetuning
and parameter-efficient finetuning with adapters. We find that mBART with
adapters performs similarly to mT5 of the same size, and NLLB can be
competitive in some cases. We also underline the importance of tuning learning
rate used for finetuning, which helps to alleviate the problem of generation in
the wrong language.

通过在一个语言上对多语言预训练语言模型（mPLM）进行微调，再在其他语言上使用它进行预测，零样本跨语言生成通常会出现使用错误语言生成的问题，本研究测试了替代的 mPLMs（如 mBART 和 NLLB），考虑到全微调和适配器参数高效微调，并发现带适配器的 mBART 的性能与相同规模的 mT5 相似，NLLB 在某些情况下具备竞争力，还强调了微调中学习率的调整对缓解错误语言生成问题的重要性。

预训练的多语言语言模型在零差生成中的实证研究

Empirical study of pretrained multilingual language models for zero-shot  cross-lingual generation

A popular approach to creating a zero-shot cross-language retrieval model is
to substitute a monolingual pretrained language model in the retrieval model
with a multilingual pretrained language model such as Multilingual BERT. This
multilingual model is fined-tuned to the retrieval task with monolingual data
such as English MS MARCO using the same training recipe as the monolingual
retrieval model used. However, such transferred models suffer from mismatches
in the languages of the input text during training and inference. In this work,
we propose transferring monolingual retrieval models using adapters, a
parameter-efficient component for a transformer network. By adding adapters
pretrained on language tasks for a specific language with task-specific
adapters, prior work has shown that the adapter-enhanced models perform better
than fine-tuning the entire model when transferring across languages in various
NLP tasks. By constructing dense retrieval models with adapters, we show that
models trained with monolingual data are more effective than fine-tuning the
entire model when transferring to a Cross Language Information Retrieval (CLIR)
setting. However, we found that the prior suggestion of replacing the language
adapters to match the target language at inference time is suboptimal for dense
retrieval models. We provide an in-depth analysis of this discrepancy between
other cross-language NLP tasks and CLIR.

本文研究了利用 adapters 实现单语言检索模型的跨语言迁移，证明了使用 adapters 的密集检索模型在语言迁移方面比整个模型 fine-tuning 更加有效。