BriefGPT.xyz
Mar, 2024
JORA:JAX 张量并行 LoRA 检索增强微调库
JORA: JAX Tensor-Parallel LoRA Library for Retrieval Augmented Fine-Tuning
HTML
PDF
Anique Tahir, Lu Cheng, Huan Liu
TL;DR
通过使用分布式训练,借助JAX的即时编译(JIT)和张量分片,我们引入了一种新的PEFT兼容的Llama-2模型微调框架,以有效管理资源,从而实现了加速微调并减少内存需求,从而显着改善了用于复杂RAG应用的LLM的可扩展性和可行性。
Abstract
The scaling of
large language models
(LLMs) for retrieval-based tasks, particularly in
retrieval augmented generation
(RAG), faces significant
me
→