We introduce CDBERT, a new learning paradigm that enhances the semantics
understanding ability of the Chinese PLMs with dictionary knowledge and
structure of Chinese characters. We name the two core modules of CDBERT as
Shuowen and Jiezi, where Shuowen refers to the process of retrieving the most
appropriate meaning from Chinese dictionaries and Jiezi refers to the process
of enhancing characters' glyph representations with structure understanding. To
facilitate dictionary understanding, we propose three pre-training tasks, i.e.,
Masked Entry Modeling, Contrastive Learning for Synonym and Antonym, and
Example Learning. We evaluate our method on both modern Chinese understanding
benchmark CLUE and ancient Chinese benchmark CCLUE. Moreover, we propose a new
polysemy discrimination task PolyMRC based on the collected dictionary of
ancient Chinese. Our paradigm demonstrates consistent improvements on previous
Chinese PLMs across all tasks. Moreover, our approach yields significant
boosting on few-shot setting of ancient Chinese understanding.

CDBERT 是一种改进了语义理解的学习范例，它结合了语汇知识和汉字结构。该方法使用 Shuowen 和 Jiezi 两个核心模块，前者用于从汉字字典中获取最合适的意义，后者用于通过结构理解增强汉字的字形表示。通过在现代汉语理解基准 CLUE 和古代汉语基准 CCLUE 上评估，我们的方法表现出对以前的汉语 PLM 的一致提高。此外，我们在收集的古代汉字字典上提出了一种新的多义词鉴别任务 PolyMRC，并在古代汉语理解的少量数据情况下获得了显著的提升。

说文解字：重新思考中文语言预训练的词典和字形

Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language  Pre-training

Pretrained language models (PLMs) have shown marvelous improvements across
various NLP tasks. Most Chinese PLMs simply treat an input text as a sequence
of characters, and completely ignore word information. Although Whole Word
Masking can alleviate this, the semantics in words is still not well
represented. In this paper, we revisit the segmentation granularity of Chinese
PLMs. We propose a mixed-granularity Chinese BERT (MigBERT) by considering both
characters and words. To achieve this, we design objective functions for
learning both character and word-level representations. We conduct extensive
experiments on various Chinese NLP tasks to evaluate existing PLMs as well as
the proposed MigBERT. Experimental results show that MigBERT achieves new SOTA
performance on all these tasks. Further analysis demonstrates that words are
semantically richer than characters. More interestingly, we show that MigBERT
also works with Japanese. Our code and model have been released
here~\footnote{this https URL}.

本文提出了一种混合粒度的中文 BERT 模型（MigBERT），通过同时考虑字符和词来学习其特征表示，并在各种中文 NLP 任务上进行了广泛的实验以评估现有的 PLMs 以及所提出的 MigBERT，实验结果表明，MigBERT 在所有这些任务中都实现了新的最佳表现，MigBERT 还能与日语一起使用。

汉语预训练语言模型中的分词粒度：字符、词语还是两者兼备？

Character, Word, or Both? Revisiting the Segmentation Granularity for Chinese Pre-trained Language Models

Pre-trained Language Models (PLMs) have achieved remarkable performance gains
across numerous downstream tasks in natural language understanding. Various
Chinese PLMs have been successively proposed for learning better Chinese
language representation. However, most current models use Chinese characters as
inputs and are not able to encode semantic information contained in Chinese
words. While recent pre-trained models incorporate both words and characters
simultaneously, they usually suffer from deficient semantic interactions and
fail to capture the semantic relation between words and characters. To address
the above issues, we propose a simple yet effective PLM CLOWER, which adopts
the Contrastive Learning Over Word and charactER representations. In
particular, CLOWER implicitly encodes the coarse-grained information (i.e.,
words) into the fine-grained representations (i.e., characters) through
contrastive learning on multi-grained information. CLOWER is of great value in
realistic scenarios since it can be easily incorporated into any existing
fine-grained based PLMs without modifying the production pipelines.Extensive
experiments conducted on a range of downstream tasks demonstrate the superior
performance of CLOWER over several state-of-the-art baselines.

本文提出一种名为 CLOWER 的基于对比学习的 pre-trained Language Model，能有效地融合中文词汇的语义信息，提高下游自然语言理解任务的性能。