Decision-making, a complicated task requiring various types of abilities,
presents an excellent framework for assessing Large Language Models (LLMs). Our
research investigates LLMs' decision-making capabilities through the lens of a
well-established field, Game Theory. We focus specifically on games that
support the participation of more than two agents simultaneously. Subsequently,
we introduce our framework, GAMA-Bench, including eight classical multi-agent
games. We design a scoring scheme to assess a model's performance in these
games quantitatively. Through GAMA-Bench, we investigate LLMs' robustness,
generalizability, and enhancement strategies. Results reveal that while GPT-3.5
shows satisfying robustness, its generalizability is relatively limited.
However, its performance can be improved through approaches such as
Chain-of-Thought. Additionally, we conduct evaluations across various LLMs and
find that GPT-4 outperforms other models on GAMA-Bench, achieving a score of
72.5. Moreover, the increasingly higher scores across the three iterations of
GPT-3.5 (0613, 1106, 0125) demonstrate marked advancements in the model's
intelligence with each update. The code and experimental results are made
publicly available via this https URL

通过 Game Theory 基于 GAMA-Bench 框架评估 LLMs 的决策能力，研究发现 GPT-3.5 在鲁棒性方面表现出色，但其概括能力相对有限，通过 Chain-of-Thought 等方法可以提高其性能；此外，评估发现 GPT-4 在 GAMA-Bench 上表现最好，得分为 72.5，而 GPT-3.5 的不断更新也标志着模型智能的显着提高。