In their seminal article on semantic vectors, Landauer and Dumain (1997)
proposed testing the quality of AI language models with a challenging
vocabulary test. We show that their Test of English as a Foreign Language
(TOEFL) test remains informative for contemporary major language models, since
none of the models was perfect and made errors on divergent items. The TOEFL
test consists of target words with four alternatives to choose from. We further
tested the models on a Yes/No test that requires distinguishing between
existing words and made-up nonwords. The models performed significantly worse
on the nonword items, in line with other observations that current major
language models provide non-existent information. The situation was worse when
we generalized the tests to Spanish. Here, most models gave
meanings/translations for the majority of random letter sequences. On the plus
side, the best models began to perform quite well, and they also pointed to
nonwords that were unknown to the test participants but can be found in
dictionaries.

通过对智能语言模型进行语言能力测试，我们发现 TOEFL 考试对于现代主要语言模型仍有指导意义。然而，这些模型在空白词汇项上表现较差，并且在西班牙语上泛化测试时情况更糟。尽管如此，最好的模型表现出色，并且能够指出字典中未知的词汇。

评估大规模语言模型的词汇测试的持续有效性

The continued usefulness of vocabulary tests for evaluating large  language models

Conducting experiments with diverse participants in their native languages
can uncover insights into culture, cognition, and language that may not be
revealed otherwise. However, conducting these experiments online makes it
difficult to validate self-reported language proficiency. Furthermore, existing
proficiency tests are small and cover only a few languages. We present an
automated pipeline to generate vocabulary tests using text from Wikipedia. Our
pipeline samples rare nouns and creates pseudowords with the same low-level
statistics. Six behavioral experiments (N=236) in six countries and eight
languages show that (a) our test can distinguish between native speakers of
closely related languages, (b) the test is reliable ($r=0.82$), and (c)
performance strongly correlates with existing tests (LexTale) and self-reports.
We further show that test accuracy is negatively correlated with the linguistic
distance between the tested and the native language. Our test, available in
eight languages, can easily be extended to other languages.

通过 Wikipedia 生成词汇测试，评估参与者的母语能力，可以在不同语言文化背景下了解文化、认知和语言的视角，通过六项行为实验测试结果表明，这个测试可以区分母语相近的语言，测试结果与现有测试成绩和个人报告结果有很强的相关性。