Words change their meaning over time as well as in different contexts. The sense-aware contextualised word embeddings (SCWEs) such as the ones produced by XL-LEXEME by fine-tuning masked langauge models (MLMs) on Word-in-Context (WiC) data attempt to encode such semantic changes of words within the contextualised word embedding (CWE) spaces. Despite the superior performance of SCWEs in contextual/temporal semantic change detection (SCD) benchmarks, it remains unclear as to how the meaning changes are encoded in the embedding space. To study this, we compare pre-trained CWEs and their fine-tuned versions on contextual and temporal semantic change benchmarks under Principal Component Analysis (PCA) and Independent Component Analysis (ICA) transformations. Our experimental results reveal several novel insights such as (a) although there exist a smaller number of axes that are responsible for semantic changes of words in the pre-trained CWE space, this information gets distributed across all dimensions when fine-tuned, and (b) in contrast to prior work studying the geometry of CWEs, we find that PCA to better represent semantic changes than ICA. Source code is available at https://github.com/LivNLP/svp-dims .

词语会随着时间和不同的语境而改变其含义。词语在上下文感知的上下文化词嵌入空间中编码语义变化，在语义变化检测基准中具有卓越性能。本研究通过使用PCA和ICA转换在预训练的词嵌入和微调后的词嵌入之间进行比较，揭示了几个新的洞察，例如信息如何在嵌入空间中分布和PCA能更好地表示语义变化。

研究上下文词嵌入维度对上下文和时间语义变化的影响