In their seminal article on semantic vectors, Landauer and Dumain (1997)
proposed testing the quality of AI language models with a challenging
vocabulary test. We show that their Test of English as a Foreign Language
(TOEFL) test remains informative for contemporary major language models, since
none of the models was perfect and made errors on divergent items. The TOEFL
test consists of target words with four alternatives to choose from. We further
tested the models on a Yes/No test that requires distinguishing between
existing words and made-up nonwords. The models performed significantly worse
on the nonword items, in line with other observations that current major
language models provide non-existent information. The situation was worse when
we generalized the tests to Spanish. Here, most models gave
meanings/translations for the majority of random letter sequences. On the plus
side, the best models began to perform quite well, and they also pointed to
nonwords that were unknown to the test participants but can be found in
dictionaries.

通过对智能语言模型进行语言能力测试，我们发现 TOEFL 考试对于现代主要语言模型仍有指导意义。然而，这些模型在空白词汇项上表现较差，并且在西班牙语上泛化测试时情况更糟。尽管如此，最好的模型表现出色，并且能够指出字典中未知的词汇。

评估大规模语言模型的词汇测试的持续有效性

The continued usefulness of vocabulary tests for evaluating large  language models

Recent popularity surrounds large AI language models due to their impressive
natural language capabilities. They contribute significantly to
language-related tasks, including prompt-based learning, making them valuable
for various specific tasks. This approach unlocks their full potential,
enhancing precision and generalization. Research communities are actively
exploring their applications, with ChatGPT receiving recognition.
Despite extensive research on large language models, their potential in
recommendation scenarios still needs to be explored. This study aims to fill
this gap by investigating ChatGPT's capabilities as a zero-shot recommender
system. Our goals include evaluating its ability to use user preferences for
recommendations, reordering existing recommendation lists, leveraging
information from similar users, and handling cold-start situations. We assess
ChatGPT's performance through comprehensive experiments using three datasets
(MovieLens Small, Last.FM, and Facebook Book).
We compare ChatGPT's performance against standard recommendation algorithms
and other large language models, such as GPT-3.5 and PaLM-2. To measure
recommendation effectiveness, we employ widely-used evaluation metrics like
Mean Average Precision (MAP), Recall, Precision, F1, normalized Discounted
Cumulative Gain (nDCG), Item Coverage, Expected Popularity Complement (EPC),
Average Coverage of Long Tail (ACLT), Average Recommendation Popularity (ARP),
and Popularity-based Ranking-based Equal Opportunity (PopREO).
Through thoroughly exploring ChatGPT's abilities in recommender systems, our
study aims to contribute to the growing body of research on the versatility and
potential applications of large language models. Our experiment code is
available on the GitHub repository:
this https URL

本研究旨在填补大型语言模型在推荐场景中的潜力研究空白，并通过实验评估 ChatGPT 在零样本推荐系统中的能力及其与标准推荐算法以及其他大型语言模型的对比表现。