The rapid evolution of large language models (LLMs) holds promise for
reforming the methodology of spatio-temporal data mining. However, current
works for evaluating the spatio-temporal understanding capability of LLMs are
somewhat limited and biased. These works either fail to incorporate the latest
language models or only focus on assessing the memorized spatio-temporal
knowledge. To address this gap, this paper dissects LLMs' capability of
spatio-temporal data into four distinct dimensions: knowledge comprehension,
spatio-temporal reasoning, accurate computation, and downstream applications.
We curate several natural language question-answer tasks for each category and
build the benchmark dataset, namely STBench, containing 13 distinct tasks and
over 60,000 QA pairs. Moreover, we have assessed the capabilities of 13 LLMs,
such as GPT-4o, Gemma and Mistral. Experimental results reveal that existing
LLMs show remarkable performance on knowledge comprehension and spatio-temporal
reasoning tasks, with potential for further enhancement on other tasks through
in-context learning, chain-of-though prompting, and fine-tuning. The code and
datasets of STBench are released on this https URL

该论文通过评估大型语言模型的时空数据理解能力，将其能力分解为知识理解、时空推理、准确计算和下游应用四个维度，并通过构建基准数据集 STBench 以及对 13 个语言模型的评估实验，揭示现有语言模型在知识理解和时空推理任务上表现出色，且通过在上下文学习、思维链提示和微调方面有进一步优化的潜力。

STBench: 大型语言模型在时空分析中的能力评估

STBench: Assessing the Ability of Large Language Models in  Spatio-Temporal Analysis

Large Language Models (LLMs) have demonstrated impressive performance on
several benchmarks. However, traditional studies do not provide formal
guarantees on the performance of LLMs. In this work, we propose a novel
certification framework for LLM, QuaCer-C, wherein we formally certify the
knowledge-comprehension capabilities of popular LLMs. Our certificates are
quantitative - they consist of high-confidence, tight bounds on the probability
that the target LLM gives the correct answer on any relevant knowledge
comprehension prompt. Our certificates for the Llama, Vicuna, and Mistral LLMs
indicate that the knowledge comprehension capability improves with an increase
in the number of parameters and that the Mistral model is less performant than
the rest in this evaluation.

提出了一种新的 QuaCer-C 证明框架来正式认证流行的 LLMs 的知识理解能力，通过高置信度的概率上界，证明 LLMs 在任何相关知识理解提示上给出正确答案的能力与参数数量的增加而提高，Mistral 模型在这个评估中表现不佳。