The growing interest in Large Language Models (LLMs) and in particular in conversational models with which users can interact has led to the development of a large number of open-source chat LLMs. These models are evaluated on a wide range of benchmarks to assess their capabilities in answering questions or solving problems on almost any possible topic or to test their ability to reason or interpret texts. Instead, the evaluation of the knowledge that these models have of the languages has received much less attention. For example, the words that they can recognize and use in different languages. In this paper, we evaluate the knowledge that open-source chat LLMs have of Spanish words by testing a sample of words in a reference dictionary. The results show that open-source chat LLMs produce incorrect meanings for an important fraction of the words and are not able to use most of the words correctly to write sentences with context. These results show how Spanish is left behind in the open-source LLM race and highlight the need to push for linguistic fairness in conversational LLMs ensuring that they provide similar performance across languages.

开源聊天型大型语言模型评估了它们对西班牙语单词的知识，结果发现它们对大部分单词的意思产生错误的理解，并且无法正确运用这些单词构建有上下文的句子，揭示了西班牙语在开源大型语言模型竞赛中的滞后性，并强调了在聊天型大型语言模型中推动语言公平性的需求。

开源会话LM不认识大部分西班牙语单词