During the fine-tuning phase of transfer learning, the pretrained vocabulary
remains unchanged, while model parameters are updated. The vocabulary generated
based on the pretrained data is suboptimal for downstream data when domain
discrepancy exists. We propose to consider the vocabulary as an optimizable
parameter, allowing us to update the vocabulary by expanding it with
domain-specific vocabulary based on a tokenization statistic. Furthermore, we
preserve the embeddings of the added words from overfitting to downstream data
by utilizing knowledge learned from a pretrained language model with a
regularization term. Our method achieved consistent performance improvements on
diverse domains (i.e., biomedical, computer science, news, and reviews).

本文提出了一种针对迁移学习微调阶段中预训练词汇表不能适用于不同领域数据的问题，通过考虑词汇表为可优化参数，利用基于分词统计的领域特定词汇表对其进行扩展和优化，同时利用预训练的语言模型知识和正则化项防止新添加单词的过拟合，实现在不同领域（如生物医学、计算机科学、新闻和评论）上的一致性性能提升。