Can Large Language Models (LLMs) simulate human behavior in complex
environments? LLMs have recently been shown to exhibit advanced reasoning
skills but much of NLP evaluation still relies on static benchmarks. Answering
this requires evaluation environments that probe strategic reasoning in
competitive, dynamic scenarios that involve long-term planning. We introduce
AucArena, a novel simulation environment for evaluating LLMs within auctions, a
setting chosen for being highly unpredictable and involving many skills related
to resource and risk management, while also being easy to evaluate. We conduct
several controlled simulations using state-of-the-art LLMs as bidding agents.
We find that through simple prompting, LLMs do indeed demonstrate many of the
skills needed for effectively engaging in auctions (e.g., managing budget,
adhering to long-term goals and priorities), skills that we find can be
sharpened by explicitly encouraging models to be adaptive and observe
strategies in past auctions. These results are significant as they show the
potential of using LLM agents to model intricate social dynamics, especially in
competitive settings. However, we also observe considerable variability in the
capabilities of individual LLMs. Notably, even our most advanced models (GPT-4)
are occasionally surpassed by heuristic baselines and human agents,
highlighting the potential for further improvements in the design of LLM agents
and the important role that our simulation environment can play in further
testing and refining agent architectures.

大型语言模型可以在复杂环境中模拟人类行为，在竞争环境中展示了高级推理技能，需要评估环境来探测战略推理、竞争动态场景中的长期规划，介绍了 AucArena 作为一个评估 LLMs 的新型模拟环境，在竞拍中证明了通过简单的提示，LLMs 确实展示了参与竞拍所需的许多技能，还发现对 LLM 代理进行自适应和观察过去竞拍策略的明确鼓励，可以提高这些技能的准确性，这些结果表明使用 LLM 代理模拟复杂社交动态的潜力，尤其在竞争环境中，但我们也观察到个体 LLMs 的能力存在相当大的变异性，值得注意的是，即使是最先进的模型（GPT-4）有时也会被启发式基准线和人类代理超越，这突显了 LLM 代理设计中进一步提高和我们的模拟环境在测试和改进代理体系结构中的重要作用。