Recent pretraining models in Chinese neglect two important aspects specific
to the Chinese language: glyph and pinyin, which carry significant syntax and
semantic information for language understanding. In this work, we propose
ChineseBERT, which incorporates both the {\it glyph} and {\it pinyin}
information of Chinese characters into language model pretraining. The glyph
embedding is obtained based on different fonts of a Chinese character, being
able to capture character semantics from the visual features, and the pinyin
embedding characterizes the pronunciation of Chinese characters, which handles
the highly prevalent heteronym phenomenon in Chinese (the same character has
different pronunciations with different meanings). Pretrained on large-scale
unlabeled Chinese corpus, the proposed ChineseBERT model yields significant
performance boost over baseline models with fewer training steps. The porpsoed
model achieves new SOTA performances on a wide range of Chinese NLP tasks,
including machine reading comprehension, natural language inference, text
classification, sentence pair matching, and competitive performances in named
entity recognition. Code and pretrained models are publicly available at
this https URL

本文提出了一种新的预训练语言模型 ChineseBERT，将汉字的字形、拼音信息融合到语言模型预训练中，该模型在多个汉语自然语言处理任务上取得了新的最佳表现。