We develop benchmarks for LLM agents that act in, learn from, and strategize in unknown environments, the specifications of which the LLM agent must learn over time from deliberate exploration. Our benchmarks consist of decision-making tasks derived from key problems in economics. To forestall saturation, the benchmark tasks are synthetically generated with scalable difficulty levels. Additionally, we propose litmus tests, a new kind of quantitative measure for LLMs and LLM agents. Unlike benchmarks, litmus tests quantify differences in character, values, and tendencies of LLMs and LLM agents, by considering their behavior when faced with tradeoffs (e.g., efficiency versus equality) where there is no objectively right or wrong behavior. Overall, our benchmarks and litmus tests assess the abilities and tendencies of LLM agents in tackling complex economic problems in diverse settings spanning procurement, scheduling, task allocation, and pricing -- applications that should grow in importance as such agents are further integrated into the economy.

本研究解决了LLM代理在未知环境中采取行动、学习与探索的能力不足的问题。作者提出了基于经济学关键问题的决策任务基准和新型量化测量方法试金石测试，利用合成生成的可扩展难度来评估LLM代理的能力和倾向。研究的主要发现是，这些基准和测试能够有效评估LLM代理在复杂经济问题中的表现，为其在经济中的应用提供了重要支持。

经济评估：在未知环境中对LLM代理的基准和试金石测试