Recent work has proposed a methodology for the systematic evaluation of
"Situated Language Understanding Agents"-agents that operate in rich linguistic
and non-linguistic contexts-through testing them in carefully constructed
interactive settings. Other recent work has argued that Large Language Models
(LLMs), if suitably set up, can be understood as (simulators of) such agents. A
connection suggests itself, which this paper explores: Can LLMs be evaluated
meaningfully by exposing them to constrained game-like settings that are built
to challenge specific capabilities? As a proof of concept, this paper
investigates five interaction settings, showing that current chat-optimised
LLMs are, to an extent, capable to follow game-play instructions. Both this
capability and the quality of the game play, measured by how well the
objectives of the different games are met, follows the development cycle, with
newer models performing better. The metrics even for the comparatively simple
example games are far from being saturated, suggesting that the proposed
instrument will remain to have diagnostic value. Our general framework for
implementing and evaluating games with LLMs is available at
this https URL

本文探索了一种方法，即在特定的游戏场景中测试大型语言模型的表现，以此来深入了解它们是否能像环境理解代理一样进行操作，涵盖了五个交互设置，并表明当前聊天优化的大型语言模型在一定程度上能够遵循游戏规则，并且用于衡量游戏表现和调查 LLM 的性能具有诊断价值。