Conventional tokenization methods for Chinese pretrained language models (PLMs) treat each character as an indivisible token (Devlin et al., 2019), which ignores the characteristics of the Chinese writing system. In this work, we comprehensively study the influences of three main factors on the Chinese tokenization for PLM: pronunciation, glyph (i.e., shape), and word boundary. Correspondingly, we propose three kinds of tokenizers: 1) SHUOWEN (meaning Talk Word), the pronunciation-based tokenizers; 2) JIEZI (meaning Solve Character), the glyph-based tokenizers; 3) Word segmented tokenizers, the tokenizers with Chinese word segmentation. To empirically compare the effectiveness of studied tokenizers, we pretrain BERT-style language models with them and evaluate the models on various downstream NLU tasks. We find that SHUOWEN and JIEZI tokenizers can generally outperform conventional single-character tokenizers, while Chinese word segmentation shows no benefit as a preprocessing step. Moreover, the proposed SHUOWEN and JIEZI tokenizers exhibit significantly better robustness in handling noisy texts. The code and pretrained models will be publicly released to facilitate linguistically informed Chinese NLP.

为了利用中文语言系统中存在的字符下的语言信息，我们提出了一种基于SubChar(即SubCharacter)的分词方法，该方法通过将每个汉字转换为一个基于其字形或发音的短序列，再基于编码后的文本进行子词分割来构建词汇表，实验结果表明SubChar的分词器相比现有分词器有两个主要优势：它们可以将输入标记为更短的序列，从而提高计算效率；以发音为基础的SubChar分词器可以将中文同音字编码为相同的音译序列并产生相同的标记化输出，从而具有抗同音错误的鲁棒性。

中文预训练语言模型的子字分词