Large language models hold significant potential for integrating various data
types, such as text documents and database records, for advanced analytics.
However, blending text and numerical data presents substantial challenges. LLMs
need to process and cross-reference entities and numbers, handle data
inconsistencies and redundancies, and develop planning capabilities such as
building a working memory for managing complex data queries. In this paper, we
introduce four novel tasks centered around sports data analytics to evaluate
the numerical reasoning and information fusion capabilities of LLMs. These
tasks involve providing LLMs with detailed, play-by-play sports game
descriptions, then challenging them with adversarial scenarios such as new game
rules, longer durations, scrambled narratives, and analyzing key statistics in
game summaries. We conduct extensive experiments on NBA and NFL games to assess
the performance of LLMs on these tasks. Our benchmark, SportsMetrics,
introduces a new mechanism for assessing LLMs' numerical reasoning and fusion
skills.

大型语言模型在整合文本文档和数据库记录等各种数据类型进行高级分析方面具有重要潜力。然而，混合文本和数值数据面临着很大的挑战。本文介绍了围绕体育数据分析的四个新颖任务，用于评估大型语言模型的数值推理和信息融合能力。这些任务涉及提供详细的场上比赛描述，并向模型提出具有挑战性的场景，例如新的比赛规则、更长的比赛时间、混乱的故事情节以及分析比赛摘要中的关键统计数据。我们在 NBA 和 NFL 比赛上进行了大量实验证明大型语言模型在这些任务上的表现。我们的基准测试系统 SportsMetrics 引入了一种评估大型语言模型数值推理和融合能力的新机制。