The ability to compare the semantic similarity between text corpora is
important in a variety of natural language processing applications. However,
standard methods for evaluating these metrics have yet to be established. We
propose a set of automatic and interpretable measures for assessing the
characteristics of corpus-level semantic similarity metrics, allowing sensible
comparison of their behavior. We demonstrate the effectiveness of our
evaluation measures in capturing fundamental characteristics by evaluating them
on a collection of classical and state-of-the-art metrics. Our measures
revealed that recently-developed metrics are becoming better in identifying
semantic distributional mismatch while classical metrics are more sensitive to
perturbations in the surface text levels.

通过提出一系列的评估度量，本文对语义相似度度量的特征进行自动和可解释的评估，从而实现了对不同语义相似度度量方法行为的合理比较。通过对经典方法和最新方法的评估，我们的度量揭示了最近开发的度量在识别语义分布不匹配方面变得更好，而经典度量则对表面文本水平的扰动更加敏感。