The recent success of distributed word representations has led to an
increased interest in analyzing the properties of their spatial distribution.
Several studies have suggested that contextualized word embedding models do not
isotropically project tokens into vector space. However, current methods
designed to measure isotropy, such as average random cosine similarity and the
partition score, have not been thoroughly analyzed and are not appropriate for
measuring isotropy. We propose IsoScore: a novel tool that quantifies the
degree to which a point cloud uniformly utilizes the ambient vector space.
Using rigorously designed tests, we demonstrate that IsoScore is the only tool
available in the literature that accurately measures how uniformly distributed
variance is across dimensions in vector space. Additionally, we use IsoScore to
challenge a number of recent conclusions in the NLP literature that have been
derived using brittle metrics of isotropy. We caution future studies from using
existing tools to measure isotropy in contextualized embedding space as
resulting conclusions will be misleading or altogether inaccurate.

通过提出量化点云普遍利用环境向量空间程度的新工具 IsoScore，该文挑战了诸多基于 NLP 文献存在问题的指标的结论。

IsoScore: 量化嵌入空间利用均匀性

IsoScore: Measuring the Uniformity of Embedding Space Utilization

Contextualized word embedding models, such as ELMo, generate meaningful
representations of words and their context. These models have been shown to
have a great impact on downstream applications. However, in many cases, the
contextualized embedding of a word changes drastically when the context is
paraphrased. As a result, the downstream model is not robust to paraphrasing
and other linguistic variations. To enhance the stability of contextualized
word embedding models, we propose an approach to retrofitting contextualized
embedding models with paraphrase contexts. Our method learns an orthogonal
transformation on the input space, which seeks to minimize the variance of word
representations on paraphrased contexts. Experiments show that the retrofitted
model significantly outperforms the original ELMo on various sentence
classification and language inference tasks.

本文章探讨了，如何通过给上下文动态变化的词嵌入模型增加同义词的语境来提高模型的稳定性，提出了一种基于改造的方法并在各种句子分类和语言推理任务中进行了实验，结果表明该方法可以有效提高模型性能。