The effectiveness of Neural Machine Translation (NMT) models largely depends
on the vocabulary used at training; small vocabularies can lead to
out-of-vocabulary problems -- large ones, to memory issues. Subword (SW)
tokenization has been successfully employed to mitigate these issues. The
choice of vocabulary and SW tokenization has a significant impact on both
training and fine-tuning an NMT model. Fine-tuning is a common practice in
optimizing an MT model with respect to new data. However, new data potentially
introduces new words (or tokens), which, if not taken into consideration, may
lead to suboptimal performance. In addition, the distribution of tokens in the
new data can differ from the distribution of the original data. As such, the
original SW tokenization model could be less suitable for the new data. Through
a systematic empirical evaluation, in this work we compare different strategies
for SW tokenization and vocabulary generation with the ultimate goal to uncover
an optimal setting for fine-tuning a domain-specific model. Furthermore, we
developed several (in-domain) models, the best of which achieves 6 BLEU points
improvement over the baseline.

本文对不同的子词分词策略、词汇生成方法及 Fine-tuning 进行了系统的实证研究，旨在找到一种最佳的 Fine-tuning 领域特定模型的设置，实验结果表明最好的模型达到了比基线模型 6 个 BLEU 分数的提升效果。

领域内翻译最佳微调的词汇和 BPE 设置的系统分析 —— 以 NMT 为例

A Systematic Analysis of Vocabulary and BPE Settings for Optimal Fine-tuning of NMT: A Case Study of In-domain Translation

Modern search systems use several large ranker models with transformer
architectures. These models require large computational resources and are not
suitable for usage on devices with limited computational resources. Knowledge
distillation is a popular compression technique that can reduce the resource
needs of such models, where a large teacher model transfers knowledge to a
small student model. To drastically reduce memory requirements and energy
consumption, we propose two extensions for a popular sentence-transformer
distillation procedure: generation of an optimal size vocabulary and
dimensionality reduction of the embedding dimension of teachers prior to
distillation. We evaluate these extensions on two different types of ranker
models. This results in extremely compressed student models whose analysis on a
test dataset shows the significance and utility of our proposed extensions.

本篇研究提出了在两种不同类型的 ranker 模型上扩展句子转换蒸馏过程的两种方法，包括生成最优尺寸的词汇表和在蒸馏之前对教师的嵌入维数进行降维。结果表明，采用这些扩展技术的学生模型具有极大的压缩程度，而且在测试数据集上表现出高度的有效性和能用性。

句向量排序模型的极端压缩：在边缘设备上实现更快的推理、更长的电池续航和更少的存储

Extreme compression of sentence-transformer ranker models: faster inference, longer battery life, and less storage on edge devices

State-of-the-art multilingual models depend on vocabularies that cover all of
the languages the model will expect to see at inference time, but the standard
methods for generating those vocabularies are not ideal for massively
multilingual applications. In this work, we introduce a novel procedure for
multilingual vocabulary generation that combines the separately trained
vocabularies of several automatically derived language clusters, thus balancing
the trade-off between cross-lingual subword sharing and language-specific
vocabularies. Our experiments show improvements across languages on key
multilingual benchmark tasks TyDi QA (+2.9 F1), XNLI (+2.1\%), and WikiAnn NER
(+2.8 F1) and factor of 8 reduction in out-of-vocabulary rate, all without
increasing the size of the model or data.

本文比较现有多语言模型词汇表生成方法，提出了一种新的多语言词汇表生成方法。实验证明，该方法能够提高多个语言上的推理性能，同时在不增加模型大小或数据的情况下，将语言识别率降低了 8 倍。