BriefGPT.xyz
Jun, 2023
通过支持预训练数据理解上下文学习
Understanding In-Context Learning via Supportive Pretraining Data
HTML
PDF
Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz...
TL;DR
通过分析预训练数据,研究了上下文学习在自然语言处理任务中的表现,并发现罕见、长尾词汇的含量较高的、具有挑战性的训练数据可以显著提高语言模型的上下文学习能力,将有助于指导未来预训练数据的构建。
Abstract
in-context learning
(ICL) improves
language models
' performance on a variety of
nlp
tasks by simply demonstrating a handful of examples at
→