Evaluation plays a significant role in modern natural language processing.
Most modern NLP benchmarks consist of arbitrary sets of tasks that neither
guarantee any generalization potential for the model once applied outside the
test set nor try to minimize the resource consumption needed for model
evaluation. This paper presents a theoretical instrument and a practical
algorithm to calculate similarity between benchmark tasks, we call this
similarity measure "Vygotsky distance". The core idea of this similarity
measure is that it is based on relative performance of the "students" on a
given task, rather that on the properties of the task itself. If two tasks are
close to each other in terms of Vygotsky distance the models tend to have
similar relative performance on them. Thus knowing Vygotsky distance between
tasks one can significantly reduce the number of evaluation tasks while
maintaining a high validation quality. Experiments on various benchmarks,
including GLUE, SuperGLUE, CLUE, and RussianSuperGLUE, demonstrate that a vast
majority of NLP benchmarks could be at least 40% smaller in terms of the tasks
included. Most importantly, Vygotsky distance could also be used for the
validation of new tasks thus increasing the generalization potential of the
future NLP models.

评估在现代自然语言处理中起着重要作用。该论文介绍了一种理论工具和一种实际算法，用于计算基准任务之间的相似性，称为 “Vygotsky 距离”。该相似性度量的核心思想是基于给定任务上 “学生” 的相对表现，而不是任务本身的特性。在维格斯基距离接近的两个任务上，模型往往具有类似的相对性能。因此，通过了解任务之间的维格斯基距离，可以显著减少评估任务的数量，同时保持高验证质量。包括 GLUE、SuperGLUE、CLUE 和 RussianSuperGLUE 在内的各种基准测试的实验表明，至少可以将大多数自然语言处理基准测试的任务数量减少 40%。最重要的是，维格斯基距离还可以用于验证新任务，从而增加未来自然语言处理模型的泛化能力。