The automatic evaluation of LLM-based agent intelligence is critical in
developing advanced LLM-based agents. Although considerable effort has been
devoted to developing human-annotated evaluation datasets, such as AlpacaEval,
existing techniques are costly, time-consuming, and lack adaptability. In this
paper, inspired by the popular language game ``Who is Spy'', we propose to use
the word guessing game to assess the intelligence performance of LLMs. Given a
word, the LLM is asked to describe the word and determine its identity (spy or
not) based on its and other players' descriptions. Ideally, an advanced agent
should possess the ability to accurately describe a given word using an
aggressive description while concurrently maximizing confusion in the
conservative description, enhancing its participation in the game. To this end,
we first develop DEEP to evaluate LLMs' expression and disguising abilities.
DEEP requires LLM to describe a word in aggressive and conservative modes. We
then introduce SpyGame, an interactive multi-agent framework designed to assess
LLMs' intelligence through participation in a competitive language-based board
game. Incorporating multi-agent interaction, SpyGame requires the target LLM to
possess linguistic skills and strategic thinking, providing a more
comprehensive evaluation of LLMs' human-like cognitive abilities and
adaptability in complex communication situations. The proposed evaluation
framework is very easy to implement. We collected words from multiple sources,
domains, and languages and used the proposed evaluation framework to conduct
experiments. Extensive experiments demonstrate that the proposed DEEP and
SpyGame effectively evaluate the capabilities of various LLMs, capturing their
ability to adapt to novel situations and engage in strategic communication.

通过使用单词猜谜游戏评估 LLM 的智能性能，本研究提出了 DEEP 和 SpyGame 两个评估框架，能够有效评估各种 LLM 的能力，并捕捉其适应新情境和战略性沟通的能力。

利用词语猜测游戏评估大型语言模型的智能

Leveraging Word Guessing Games to Assess the Intelligence of Large  Language Models

In this paper, we describe adaptation of a simple word guessing game that
occupied the hearts and minds of people around the world. There are versions
for all three Baltic countries and even several versions of each. We
specifically pay attention to the Latvian version and look into how people form
their guesses given any already uncovered hints. The paper analyses guess
patterns, easy and difficult word characteristics, and player behaviour and
response.

描绘了一个简单的单词猜谜游戏的适应性，特别是拉脱维亚版本，分析人们如何根据已揭示的提示形成猜测模式，以及参与者的行为和反应。

人们在运用词汇方面的掌握能力如何？拉脱维亚 Wordle 游戏分析

How Masterly Are People at Playing with Their Vocabulary? Analysis of the Wordle Game for Latvian

Wordle, a word guessing game rose to global popularity in the January of
2022. The goal of the game is to guess a five-letter English word within six
tries. Each try provides the player with hints by means of colour changing
tiles which inform whether or not a given character is part of the solution as
well as, in cases where it is part of the solution, whether or not it is in the
correct placement. Numerous attempts have been made to find the best starting
word and best strategy to solve the daily wordle. This study uses character
statistics of five-letter words to determine the best three starting words.

该研究使用五字母单词的字符统计学信息，确定了最佳的三个起始单词，以便解决每日的 Wordle 游戏。