Large, multilingual language models exhibit surprisingly good zero- or
few-shot machine translation capabilities, despite having never seen the
intentionally-included translation examples provided to typical neural
translation systems. We investigate the role of incidental bilingualism -- the
unintentional consumption of bilingual signals, including translation examples
-- in explaining the translation capabilities of large language models, taking
the Pathways Language Model (PaLM) as a case study. We introduce a mixed-method
approach to measure and understand incidental bilingualism at scale. We show
that PaLM is exposed to over 30 million translation pairs across at least 44
languages. Furthermore, the amount of incidental bilingual content is highly
correlated with the amount of monolingual in-language content for non-English
languages. We relate incidental bilingual content to zero-shot prompts and show
that it can be used to mine new prompts to improve PaLM's out-of-English
zero-shot translation quality. Finally, in a series of small-scale ablations,
we show that its presence has a substantial impact on translation capabilities,
although this impact diminishes with model scale.

本文通过研究 Pathways Language Model，发现大型语言模型具有出色的零翻译能力，并探讨无意中的双语翻译在解释大型语言模型的翻译能力方面的作用，它可以用于挖掘新的提示来提高 PaLM 的零翻译质量。

在大数据中寻找细微之针：偶然双语对 PaLM 翻译能力的作用

Searching for Needles in a Haystack: On the Role of Incidental  Bilingualism in PaLM's Translation Capability

Both image-caption pairs and translation pairs provide the means to learn
deep representations of and connections between languages. We use both types of
pairs in MURAL (MUltimodal, MUltitask Representations Across Languages), a dual
encoder that solves two tasks: 1) image-text matching and 2) translation pair
matching. By incorporating billions of translation pairs, MURAL extends ALIGN
(Jia et al. PMLR'21)--a state-of-the-art dual encoder learned from 1.8 billion
noisy image-text pairs. When using the same encoders, MURAL's performance
matches or exceeds ALIGN's cross-modal retrieval performance on well-resourced
languages across several datasets. More importantly, it considerably improves
performance on under-resourced languages, showing that text-text learning can
overcome a paucity of image-caption examples for these languages. On the
Wikipedia Image-Text dataset, for example, MURAL-base improves zero-shot mean
recall by 8.1% on average for eight under-resourced languages and by 6.8% on
average when fine-tuning. We additionally show that MURAL's text
representations cluster not only with respect to genealogical connections but
also based on areal linguistics, such as the Balkan Sprachbund.

本论文介绍了一种名为 MURAL 的双编码器，可以利用翻译对和图像 - 文本对从多个角度进行语言表示学习，并且可以通过对少量图像 - 文本样例的学习来改善对欠发达语言的性能。