BriefGPT.xyz
Oct, 2020
低资源领域适应的多阶段预训练
Multi-Stage Pre-training for Low-Resource Domain Adaptation
HTML
PDF
Rong Zhang, Revanth Gangi Reddy, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto...
TL;DR
本文研究使用预训练语言模型的转移学习、领域特定术语扩展词汇表以及利用未标记数据结构创造辅助合成任务等方法，在IT领域的三个任务中，采用逐步应用的策略在预先训练的Roberta-large LM上显示出显着的性能提升。
Abstract
transfer learning
techniques are particularly useful in
nlp
tasks where a sizable amount of high-quality annotated data is difficult to obtain. Current approaches directly adapt a pre-trained language model (LM)
→