Contextual word representation models have shown massive improvements on a multitude of NLP tasks, yet their word sense disambiguation capabilities remain poorly explained. To address this gap, we assess whether contextual word representations extracted from deep pretrained language models create distinguishable representations for different senses of a given word. We analyze the representation geometry and find that most layers of deep pretrained language models create highly anisotropic representations, pointing towards the existence of representation degeneration problem in contextual word representations. After accounting for anisotropy, our study further reveals that there is variability in sense learning capabilities across different language models. Finally, we propose LASeR, a 'Low Anisotropy Sense Retrofitting' approach that renders off-the-shelf representations isotropic and semantically more meaningful, resolving the representation degeneration problem as a post-processing step, and conducting sense-enrichment of contextualized representations extracted from deep neural language models.

研究了上下文单词表示模型的词义消歧能力，发现多数深度预训练语言模型的上下文单词表示在几何结构上高度异性化，并存在表示退化问题，提出了一种低异性度词义修正方法（LASeR），以解决上下文单词表示的表示退化问题。

降低各向异性感知的后期处理（LASeR）：朝向各向同性且感知丰富的表示