Natural language processing of Low-Resource Languages (LRL) is often
challenged by the lack of data. Therefore, achieving accurate machine
translation (MT) in a low-resource environment is a real problem that requires
practical solutions. Research in multilingual models have shown that some LRLs
can be handled with such models. However, their large size and computational
needs make their use in constrained environments (e.g., mobile/IoT devices or
limited/old servers) impractical. In this paper, we address this problem by
leveraging the power of large multilingual MT models using knowledge
distillation. Knowledge distillation can transfer knowledge from a large and
complex teacher model to a simpler and smaller student model without losing
much in performance. We also make use of high-resource languages that are
related or share the same linguistic root as the target LRL. For our
evaluation, we consider Luxembourgish as the LRL that shares some roots and
properties with German. We build multiple resource-efficient models based on
German, knowledge distillation from the multilingual No Language Left Behind
(NLLB) model, and pseudo-translation. We find that our efficient models are
more than 30\% faster and perform only 4\% lower compared to the large
state-of-the-art NLLB model.

本文介绍了一种使用知识蒸馏技术以及基于高资源语言的资源有效模型实现低资源语言机器翻译的方法。在以卢森堡语为例子的实验中，资源有效模型相比于大型 NLLB 模型速度提高了 30％以上且性能仅下降了 4％。

Letz Translate: 用于卢森堡语的低资源机器翻译

Letz Translate: Low-Resource Machine Translation for Luxembourgish

Pre-trained Transformer-based models have achieved state-of-the-art
performance for various Natural Language Processing (NLP) tasks. However, these
models often have billions of parameters, and, thus, are too resource-hungry
and computation-intensive to suit low-capability devices or applications with
strict latency requirements. One potential remedy for this is model
compression, which has attracted a lot of research attention. Here, we
summarize the research in compressing Transformers, focusing on the especially
popular BERT model. In particular, we survey the state of the art in
compression for BERT, we clarify the current best practices for compressing
large-scale Transformer models, and we provide insights into the workings of
various methods. Our categorization and analysis also shed light on promising
future research directions for achieving lightweight, accurate, and generic NLP
models.

本文总结了压缩预训练 Transformer 模型的研究进展，尤其关注流行的 BERT 模型的最佳压缩实践和方法，提出未来发展的方向，以达到轻量、高准确性和通用性的自然语言处理模型。