Large Language Models (LLMs) have excelled in various tasks but perform
better in high-resource scenarios, which presents challenges in low-resource
scenarios. Data scarcity and the inherent difficulty of adapting LLMs to
specific tasks compound the challenge. To address the twin hurdles, we
introduce \textbf{Leverage Learning}. We present a streamlined implement of
this methodology called Token-Efficient Leverage Learning (TELL). TELL
showcases the potential of Leverage Learning, demonstrating effectiveness
across various LLMs and low-resource tasks, ranging from $10^4$ to $10^6$
tokens. It reduces task data requirements by up to nearly an order of magnitude
compared to conventional Supervised Fine-Tuning (SFT) while delivering
competitive performance. With the same amount of task data, TELL leads in
improving task performance compared to SFT. We discuss the mechanism of
Leverage Learning, suggesting it aligns with quantization hypothesis and
explore its promising potential through empirical testing.

通过引入 “Leverage Learning” 方法，我们实现了一种简化的实施方式，称为 “Token-Efficient Leverage Learning (TELL)”。TELL 展示了 Leverage Learning 的潜力，并在各种 LLMs 和低资源任务中展示了有效性，任务数据需求降低了近一个数量级，并且在与传统的监督微调 (SFT) 相比表现出有竞争力的性能。我们讨论了 Leverage Learning 的机制，提出它与量化假设相一致，并通过经验测试探索了它的潜在潜力。

大型语言模型中高效利用学习

Token-Efficient Leverage Learning in Large Language Models

Sparsely gated Mixture of Experts (MoE) models have been shown to be a
compute-efficient method to scale model capacity for multilingual machine
translation. However, for low-resource tasks, MoE models severely over-fit. We
show effective regularization strategies, namely dropout techniques for MoE
layers in EOM and FOM, Conditional MoE Routing and Curriculum Learning methods
that prevent over-fitting and improve the performance of MoE models on
low-resource tasks without adversely affecting high-resource tasks. On a
massively multilingual machine translation benchmark, our strategies result in
about +1 chrF++ improvement in very low resource language pairs. We perform an
extensive analysis of the learned MoE routing to better understand the impact
of our regularization methods and how we can improve them.

通过引入 dropout 和条件化的 MoE 路由和课程学习技术，本研究针对低资源任务下 MoE 模型的严重过拟合问题，提出有效的正则化策略，成功提高了 MoE 模型在低资源任务中的性能，并在大规模多语种机器翻译基准测试中取得了巨大的改进。

多语言机器翻译中修复低资源语言的 MoE 过拟合问题

Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation

Data augmentation is one of the most effective ways to make end-to-end
automatic speech recognition (ASR) perform close to the conventional hybrid
approach, especially when dealing with low-resource tasks. Using recent
advances in speech synthesis (text-to-speech, or TTS), we build our TTS system
on an ASR training database and then extend the data with synthesized speech to
train a recognition model. We argue that, when the training data amount is
relatively low, this approach can allow an end-to-end model to reach hybrid
systems' quality. For an artificial low-to-medium-resource setup, we compare
the proposed augmentation with the semi-supervised learning technique. We also
investigate the influence of vocoder usage on final ASR performance by
comparing Griffin-Lim algorithm with our modified LPCNet. When applied with an
external language model, our approach outperforms a semi-supervised setup for
LibriSpeech test-clean and only 33% worse than a comparable supervised setup.
Our system establishes a competitive result for end-to-end ASR trained on
LibriSpeech train-clean-100 set with WER 4.3% for test-clean and 13.5% for
test-other.

采用数据增强和 TTS 技术，对 ASR 的训练数据进行扩充，并通过集成语言模型，在 LibriSpeech 数据上建立 end-to-end 模型，相对于半监督技术的效果更好。