Large language models (LLMs) with powerful generalization ability has been
widely used in many domains. A systematic and reliable evaluation of LLMs is a
crucial step in their development and applications, especially for specific
professional fields. In the urban domain, there have been some early
explorations about the usability of LLMs, but a systematic and scalable
evaluation benchmark is still lacking. The challenge in constructing a
systematic evaluation benchmark for the urban domain lies in the diversity of
data and scenarios, as well as the complex and dynamic nature of cities. In
this paper, we propose CityBench, an interactive simulator based evaluation
platform, as the first systematic evaluation benchmark for the capability of
LLMs for urban domain. First, we build CitySim to integrate the multi-source
data and simulate fine-grained urban dynamics. Based on CitySim, we design 7
tasks in 2 categories of perception-understanding and decision-making group to
evaluate the capability of LLMs as city-scale world model for urban domain. Due
to the flexibility and ease-of-use of CitySim, our evaluation platform
CityBench can be easily extended to any city in the world. We evaluate 13
well-known LLMs including open source LLMs and commercial LLMs in 13 cities
around the world. Extensive experiments demonstrate the scalability and
effectiveness of proposed CityBench and shed lights for the future development
of LLMs in urban domain. The dataset, benchmark and source codes are openly
accessible to the research community via
this https URL

在这篇论文中，我们提出了 CityBench 作为第一个用于评估大规模语言模型在城市领域能力的系统性评估基准，通过构建 CitySim 来整合多源数据并模拟细粒度的城市动态，设计了 7 个任务用于评估 LLMs 作为城市规模世界模型在感知理解和决策制定方面的能力，在 13 个城市的 13 个知名 LLMs 上进行了广泛实验，结果表明 CityBench 的可扩展性和效果，并对未来城市领域 LLMs 的发展提供了启示。