Multilingual Language Models (MLLMs) exhibit robust cross-lingual transfer
capabilities, or the ability to leverage information acquired in a source
language and apply it to a target language. These capabilities find practical
applications in well-established Natural Language Processing (NLP) tasks such
as Named Entity Recognition (NER). This study aims to investigate the
effectiveness of a source language when applied to a target language,
particularly in the context of perturbing the input test set. We evaluate on 13
pairs of languages, each including one high-resource language (HRL) and one
low-resource language (LRL) with a geographic, genetic, or borrowing
relationship. We evaluate two well-known MLLMs--MBERT and XLM-R--on these
pairs, in native LRL and cross-lingual transfer settings, in two tasks, under a
set of different perturbations. Our findings indicate that NER cross-lingual
transfer depends largely on the overlap of entity chunks. If a source and
target language have more entities in common, the transfer ability is stronger.
Models using cross-lingual transfer also appear to be somewhat more robust to
certain perturbations of the input, perhaps indicating an ability to leverage
stronger representations derived from the HRL. Our research provides valuable
insights into cross-lingual transfer and its implications for NLP applications,
and underscores the need to consider linguistic nuances and potential
limitations when employing MLLMs across distinct languages.

多语言语言模型（MLLMs）展现了强大的跨语言转移能力，本研究旨在调查源语言应用于目标语言的效果，特别是在扰动输入测试集的情况下，发现命名实体识别的跨语言转移主要取决于实体块的重叠，研究结果提供了宝贵的跨语言转移见解并强调了在跨不同语言时考虑语言细微差异和潜在限制的必要性。

对抗性数据集上跨语言迁移至资源较少的语言的鲁棒性

Cross-Lingual Transfer Robustness to Lower-Resource Languages on  Adversarial Datasets

Using model weights pretrained on a high-resource language as a warm start
can reduce the need for data and compute to obtain high-quality language models
in low-resource languages. To accommodate the new language, the pretrained
vocabulary and embeddings need to be adapted. Previous work on embedding
initialization for such adapted vocabularies has mostly focused on monolingual
source models. In this paper, we investigate the multilingual source model
setting and propose FOCUS - Fast Overlapping Token Combinations Using
Sparsemax, a novel embedding initialization method that outperforms previous
work when adapting XLM-R. FOCUS represents newly added tokens as combinations
of tokens in the overlap of the pretrained and new vocabularies. The
overlapping tokens are selected based on semantic similarity in an auxiliary
token embedding space. Our implementation of FOCUS is publicly available on
GitHub.

使用高资源语言预训练的模型权重作为启动可以降低获取低资源语言高质量语言模型所需的数据和计算量，为了适应新语言，需要调整预先训练的词汇表和嵌入。本文研究了多语言源模型设置，提出了 FOCUS - Fast Overlapping Token Combinations Using Sparsemax 一种嵌入初始化方法，该方法在适应 XLM-R 时优于先前的方法，FOCUS 将新添加的令牌表示为预训练和新词汇表重叠的令牌组合。重叠的令牌是基于辅助令牌嵌入空间中的语义相似性选择的。

专注：有效的嵌入初始化策略用于针对单一语言进行预训练的多语言模型优化

FOCUS: Effective Embedding Initialization for Specializing Pretrained  Multilingual Models on a Single Language

The objective of this work is to explore the learning of visually grounded
speech models (VGS) from multilingual perspective. Bilingual VGS models are
generally trained with an equal number of spoken captions from both languages.
However, in reality, there can be an imbalance among the languages for the
available spoken captions. Our key contribution in this work is to leverage the
power of a high-resource language in a bilingual visually grounded speech model
to improve the performance of a low-resource language. We introduce two methods
to distill the knowledge of high-resource language into low-resource languages:
(1) incorporating a strong pre-trained high-resource language encoder and (2)
using semantically similar spoken captions. Our experiments show that combining
these two approaches effectively enables the low-resource language to surpass
the performances of monolingual and bilingual counterparts for cross-modal
retrieval tasks.

本研究旨在从多语言角度探讨视觉引导语音模型（VGS）的学习。研究发现，将高资源语言的知识转化为低资源语言的知识可以提高跨模态检索任务中低资源语言的性能，对此，文章提出了两种方法：（1）使用强大的预训练高资源语言编码器和（2）使用语义相似的口语字幕。通过实验证明，这两种方法结合起来可以有效地使低资源语言的表现超越单语和双语对应物。