BriefGPT.xyz
Oct, 2024
公平语言模型悖论
The Fair Language Model Paradox
HTML
PDF
Andrea Pinto, Tomer Galanti, Randall Balestriero
TL;DR
本研究探讨了大型语言模型在训练过程中存在的令牌级别的性能偏差,传统评估方法未能揭示这种细微的偏差。作者创新性地指出,权重衰减法在稳定训练的同时,实际上对低频令牌产生了不成比例的贬值,强调了需要新型正则化技术以确保训练过程的公平性。
Abstract
Large
Language Models
(LLMs) are widely deployed in real-world applications, yet little is known about their
Training Dynamics
at the token level. Evaluation typically relies on aggregated training loss, measured
→