The success of large pretrained language models (LMs) such as BERT and
RoBERTa has sparked interest in probing their representations, in order to
unveil what types of knowledge they implicitly capture. While prior research
focused on morphosyntactic, semantic, and world knowledge, it remains unclear
to which extent LMs also derive lexical type-level knowledge from words in
context. In this work, we present a systematic empirical analysis across six
typologically diverse languages and five different lexical tasks, addressing
the following questions: 1) How do different lexical knowledge extraction
strategies (monolingual versus multilingual source LM, out-of-context versus
in-context encoding, inclusion of special tokens, and layer-wise averaging)
impact performance? How consistent are the observed effects across tasks and
languages? 2) Is lexical knowledge stored in few parameters, or is it scattered
throughout the network? 3) How do these representations fare against
traditional static word vectors in lexical tasks? 4) Does the lexical
information emerging from independently trained monolingual LMs display latent
similarities? Our main results indicate patterns and best practices that hold
universally, but also point to prominent variations across languages and tasks.
Moreover, we validate the claim that lower Transformer layers carry more
type-level lexical knowledge, but also show that this knowledge is distributed
across multiple layers.

本研究针对 6 种不同语言和 5 种不同词汇任务，对单语和多语言源 LM、无上下文编码和有上下文编码、特殊标记的包含和分层平均等不同词汇知识提取策略的影响进行了系统的实证分析，并验证了较低变压器层携带更多类型级词汇知识的主张。

探究预训练语言模型的词汇语义

Probing Pretrained Language Models for Lexical Semantics

Current approaches to computational lexicology in language technology are
knowledge-based (competence-oriented) and try to abstract away from specific
formalisms, domains, and applications. This results in severe complexity,
acquisition and reusability bottlenecks. As an alternative, we propose a
particular performance-oriented approach to Natural Language Processing based
on automatic memory-based learning of linguistic (lexical) tasks. The
consequences of the approach for computational lexicology are discussed, and
the application of the approach on a number of lexical acquisition and
disambiguation tasks in phonology, morphology and syntax is described.

提出了一种基于自动内存学习语言（词汇）任务的自然语言处理的性能导向方法，讨论了其对计算词汇学的影响，并描述了该方法在音韵学、形态学和句法中对许多词汇获取和消歧任务的应用。