Latest instruction-tuned large language models (LLM) show great results on
various tasks, however, they often face performance degradation for non-English
input. There is evidence that the reason lies in inefficient tokenization
caused by low language representation in pre-training data which hinders the
comprehension of non-English instructions, limiting the potential of target
language instruction-tuning. In this work we investigate the possibility of
addressing the issue with vocabulary substitution in the context of LLaMa
Russian language adaptation. We explore three variants of vocabulary adaptation
and test their performance on Saiga instruction-tuning and fine-tuning on
Russian Super Glue benchmark. The results of automatic evaluation show that
vocabulary substitution not only improves the model's quality in Russian but
also accelerates fine-tuning (35%) and inference (up to 60%) while reducing
memory consumption. Additional human evaluation of the instruction-tuned models
demonstrates that models with Russian-adapted vocabulary generate answers with
higher user preference than the original Saiga-LLaMa model.

通过词汇替换来解决 LLaMa 俄语适应问题，提高模型质量并加速微调和推理，同时减少内存消耗。

令牌化对 LLaMa 俄文适应性的影响

Impact of Tokenization on LLaMa Russian Adaptation

Spellchecking is one of the most fundamental and widely used search features.
Correcting incorrectly spelled user queries not only enhances the user
experience but is expected by the user. However, most widely available
spellchecking solutions are either lower accuracy than state-of-the-art
solutions or too slow to be used for search use cases where latency is a key
requirement. Furthermore, most innovative recent architectures focus on English
and are not trained in a multilingual fashion and are trained for spell
correction in longer text, which is a different paradigm from spell correction
for user queries, where context is sparse (most queries are 1-2 words long).
Finally, since most enterprises have unique vocabularies such as product names,
off-the-shelf spelling solutions fall short of users' needs. In this work, we
build a multilingual spellchecker that is extremely fast and scalable and that
adapts its vocabulary and hence speller output based on a specific product's
needs. Furthermore, our speller out-performs general purpose spellers by a wide
margin on in-domain datasets. Our multilingual speller is used in search in
Adobe products, powering autocomplete in various applications.

该论文介绍了一种多语种的拼写检查器，它可以根据特定产品的需求适应其词汇，并在领域特定的数据集上性能优异，可以用于搜索和为各种应用程序提供自动完成功能。

面向用户查询的上下文多语言拼写检查器

Contextual Multilingual Spellchecker for User Queries

We present IndoBERTweet, the first large-scale pretrained model for
Indonesian Twitter that is trained by extending a monolingually-trained
Indonesian BERT model with additive domain-specific vocabulary. We focus in
particular on efficient model adaptation under vocabulary mismatch, and
benchmark different ways of initializing the BERT embedding layer for new word
types. We find that initializing with the average BERT subword embedding makes
pretraining five times faster, and is more effective than proposed methods for
vocabulary adaptation in terms of extrinsic evaluation over seven Twitter-based
datasets.

本研究介绍了 IndoBERTweet，这是第一个大规模预训练的印度尼西亚语 Twitter 模型，它通过扩展单语训练的印度尼西亚 BERT 模型来训练，并使用了特定领域的词汇表。我们特别关注词汇不匹配下的有效模型适应性，并基准测试了不同的 BERT 嵌入层初始化方式，发现用平均 BERT 子词嵌入初始化可以使预训练速度提高五倍，并且在七个基于 Twitter 的数据集上使用外部评估比提出的词汇自适应方法更有效。

IndoBERTweet：印度尼西亚推特预训练语言模型，具有效领域特定词汇初始化

IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with  Effective Domain-Specific Vocabulary Initialization

We propose a straightforward vocabulary adaptation scheme to extend the
language capacity of multilingual machine translation models, paving the way
towards efficient continual learning for multilingual machine translation. Our
approach is suitable for large-scale datasets, applies to distant languages
with unseen scripts, incurs only minor degradation on the translation
performance for the original language pairs and provides competitive
performance even in the case where we only possess monolingual data for the new
languages.

本研究提出了一种简单的词汇适应方案，以扩大多语言机器翻译模型的语言容量，为多语言机器翻译的高效连续学习铺平了道路。我们的方法适用于大规模数据集，适用于未见过脚本的远程语言，在原始语言对的翻译性能上只有轻微的降低，在仅拥有新语言的单语数据的情况下也提供了具有竞争力的性能。