Recent researches show that pre-trained models (PTMs) are beneficial to
Chinese Word Segmentation (CWS). However, PTMs used in previous works usually
adopt language modeling as pre-training tasks, lacking task-specific prior
segmentation knowledge and ignoring the discrepancy between pre-training tasks
and downstream CWS tasks. In this paper, we propose a CWS-specific pre-trained
model METASEG, which employs a unified architecture and incorporates meta
learning algorithm into a multi-criteria pre-training task. Empirical results
show that METASEG could utilize common prior segmentation knowledge from
different existing criteria and alleviate the discrepancy between pre-trained
models and downstream CWS tasks. Besides, METASEG can achieve new
state-of-the-art performance on twelve widely-used CWS datasets and
significantly improve model performance in low-resource settings.

本研究发现，先前使用的预训练模型往往采用语言建模作为预训练任务，缺少任务特定的分词先验知识并忽略预训练任务与下游分词任务之间的差异。为此，我们提出了一种 CWS 特定的预训练模型 METASEG，它采用统一架构并将元学习算法结合到多标准预训练任务中。实验证明，METASEG 可以利用不同现有标准的常见分词先验知识，并减轻预训练模型与下游 CWS 任务之间的差异。此外，METASEG 可以在 12 个广泛使用的 CWS 数据集上实现新的最佳性能，并显著提高低资源环境下的模型性能。