We introduce second-order vector representations of words, induced from nearest neighborhood topological features in pre-trained contextual word embeddings. We then analyze the effects of using second-order embeddings as input features in two deep natural language processing models, for named entity recognition and recognizing textual entailment, as well as a linear model for paraphrase recognition. Surprisingly, we find that nearest neighbor information alone is sufficient to capture most of the performance benefits derived from using pre-trained word embeddings. Furthermore, second-order embeddings are able to handle highly heterogeneous data better than first-order representations, though at the cost of some specificity. Additionally, augmenting contextual embeddings with second-order information further improves model performance in some cases. Due to variance in the random initializations of word embeddings, utilizing nearest neighbor features from multiple first-order embedding samples can also contribute to downstream performance gains. Finally, we identify intriguing characteristics of second-order embedding spaces for further research, including much higher density and different semantic interpretations of cosine similarity.

使用预训练的上下文词嵌入中的最近邻拓扑特征，我们介绍了二阶向量词表示法，并研究其在两种自然语言处理模型以及一个线性模型中的性能。具有更高的密度和不同的余弦相似性语义解释，二阶词嵌入为处理异质数据提供了更好的解决方案，此外，它们可以进一步提高模型性能。

最近邻拓扑特征的二阶词嵌入