BriefGPT.xyz
Jul, 2024
DDK:为高效大语言模型提炼领域知识
DDK: Distilling Domain Knowledge for Efficient Large Language Models
HTML
PDF
Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que...
TL;DR
本研究针对大语言模型在计算和存储上的高需求,提出了DDK框架,用于改进学生模型通过自教师模型的知识蒸馏过程。该方法动态调整蒸馏数据集的组成,以平衡教师和学生模型在不同领域的性能差异,显著提升了学生模型的性能,并超过了现有的方法。
Abstract
Despite the advanced intelligence abilities of
Large Language Models
(LLMs) in various applications, they still face significant computational and storage demands.
Knowledge Distillation
(KD) has emerged as an ef
→