Contextualized representations give significantly improved results for a wide
range of NLP tasks. Much work has been dedicated to analyzing the features
captured by representative models such as BERT. Existing work finds that
syntactic, semantic and word sense knowledge are encoded in BERT. However,
little work has investigated word features for character-based languages such
as Chinese. We investigate Chinese BERT using both attention weight
distribution statistics and probing tasks, finding that (1) word information is
captured by BERT; (2) word-level features are mostly in the middle
representation layers; (3) downstream tasks make different use of word features
in BERT, with POS tagging and chunking relying the most on word features, and
natural language inference relying the least on such features.

本文探究了中文 BERT 在注意力权重分布统计和探测任务方面的表现，发现其捕捉了单词信息，其中单词级别特征主要集中在中间表示层，在文本理解等下游任务中，词特征以不同的方式被应用。

中文 BERT 是否编码单词结构？

Does Chinese BERT Encode Word Structure?

The pioneering research of G. K. Zipf on the relationship between word
frequency and other word features led to the formulation of various linguistic
laws. The most popular is Zipf's law for word frequencies. Here we focus on two
laws that have been studied less intensively: the meaning-frequency law, i.e.
the tendency of more frequent words to be more polysemous, and the law of
abbreviation, i.e. the tendency of more frequent words to be shorter. In a
previous work, we tested the robustness of these Zipfian laws for English,
roughly measuring word length in number of characters and distinguishing adult
from child speech. In the present article, we extend our study to other
languages (Dutch and Spanish) and introduce two additional measures of length:
syllabic length and phonemic length. Our correlation analysis indicates that
both the meaning-frequency law and the law of abbreviation hold overall in all
the analyzed languages.

本文通过在英语、荷兰语和西班牙语三种语言中，以音节长度和音素长度两种新的方式来测量单词的长度，证明了在所有语言中，单词频率与多义性和缩写的趋势存在相关性。