Similarity measures are a vital tool for understanding how language models
represent and process language. Standard representational similarity measures
such as cosine similarity and Euclidean distance have been successfully used in
static word embedding models to understand how words cluster in semantic space.
Recently, these measures have been applied to embeddings from contextualized
models such as BERT and GPT-2. In this work, we call into question the
informativity of such measures for contextualized language models. We find that
a small number of rogue dimensions, often just 1-3, dominate these measures.
Moreover, we find a striking mismatch between the dimensions that dominate
similarity measures and those which are important to the behavior of the model.
We show that simple postprocessing techniques such as standardization are able
to correct for rogue dimensions and reveal underlying representational quality.
We argue that accounting for rogue dimensions is essential for any
similarity-based analysis of contextual language models.

通过标准化技术等方法探讨相似度测量在 BERT 和 GPT-2 等上下文鉴别模型中的代表性和准确性。我们发现一些不代表整体性质的维度会干扰这些测量，并且这些测量值并不一定代表模型的真实行为。因此，我们认为分析这些模型中基于相似性的操作需要解决这些问题。

变形器语言模型中的流浪维度掩盖了表征质量

All Bark and No Bite: Rogue Dimensions in Transformer Language Models  Obscure Representational Quality

The ability to correctly model distinct meanings of a word is crucial for the
effectiveness of semantic representation techniques. However, most existing
evaluation benchmarks for assessing this criterion are tied to sense
inventories (usually WordNet), restricting their usage to a small subset of
knowledge-based representation techniques. The Word-in-Context dataset (WiC)
addresses the dependence on sense inventories by reformulating the standard
disambiguation task as a binary classification problem; but, it is limited to
the English language. We put forward a large multilingual benchmark, XL-WiC,
featuring gold standards in 12 new languages from varied language families and
with different degrees of resource availability, opening room for evaluation
scenarios such as zero-shot cross-lingual transfer. We perform a series of
experiments to determine the reliability of the datasets and to set performance
baselines for several recent contextualized multilingual models. Experimental
results show that even when no tagged instances are available for a target
language, models trained solely on the English data can attain competitive
performance in the task of distinguishing different meanings of a word, even
for distant languages. XL-WiC is available at
this https URL

提出了一种多语言的词义消歧基准测试库（称为 XL-WiC），用于评估具有不同语言背景和资源可用性的模型，即使模型仅在英语数据上进行训练，仍能在跨语言学习中达到良好的性能。