Large Language Models (LLMs) have excelled in various tasks but perform
better in high-resource scenarios, which presents challenges in low-resource
scenarios. Data scarcity and the inherent difficulty of adapting LLMs to
specific tasks compound the challenge. To address the twin hurdles, we
introduce \textbf{Leverage Learning}. We present a streamlined implement of
this methodology called Token-Efficient Leverage Learning (TELL). TELL
showcases the potential of Leverage Learning, demonstrating effectiveness
across various LLMs and low-resource tasks, ranging from $10^4$ to $10^6$
tokens. It reduces task data requirements by up to nearly an order of magnitude
compared to conventional Supervised Fine-Tuning (SFT) while delivering
competitive performance. With the same amount of task data, TELL leads in
improving task performance compared to SFT. We discuss the mechanism of
Leverage Learning, suggesting it aligns with quantization hypothesis and
explore its promising potential through empirical testing.

通过引入 “Leverage Learning” 方法，我们实现了一种简化的实施方式，称为 “Token-Efficient Leverage Learning (TELL)”。TELL 展示了 Leverage Learning 的潜力，并在各种 LLMs 和低资源任务中展示了有效性，任务数据需求降低了近一个数量级，并且在与传统的监督微调 (SFT) 相比表现出有竞争力的性能。我们讨论了 Leverage Learning 的机制，提出它与量化假设相一致，并通过经验测试探索了它的潜在潜力。