This paper explores the cutting-edge Large Language Model with analytical
reasoning on sports. Our analytical reasoning embodies the tasks of letting
large language models count how many points each team scores in a quarter in
the NBA and NFL games. Our major discoveries are in two folds. Firstly, we find
among all the models we employed, GPT-4 stands out in effectiveness, followed
by Claude-2.1, with GPT-3.5, Gemini-Pro, and Llama-2-70b lagging behind.
Specifically, we compare three different prompting techniques and a
divide-and-conquer approach, we find that the latter was the most effective.
Our divide-and-conquer approach breaks down play-by-play data into smaller,
more manageable segments, solves each piece individually, and then aggregates
them together. Besides the divide-and-conquer approach, we also explore the
Chain of Thought (CoT) strategy, which markedly improves outcomes for certain
models, notably GPT-4 and Claude-2.1, with their accuracy rates increasing
significantly. However, the CoT strategy has negligible or even detrimental
effects on the performance of other models like GPT-3.5 and Gemini-Pro.
Secondly, to our surprise, we observe that most models, including GPT-4,
struggle to accurately count the total scores for NBA quarters despite showing
strong performance in counting NFL quarter scores. This leads us to further
investigate the factors that impact the complexity of analytical reasoning
tasks with extensive experiments, through which we conclude that task
complexity depends on the length of context, the information density, and the
presence of related information. Our research provides valuable insights into
the complexity of analytical reasoning tasks and potential directions for
developing future large language models.

本文探讨了在体育领域中的先进大型语言模型与分析推理能力。我们的分析推理涉及了在 NBA 和 NFL 比赛中，大型语言模型计算每个队伍在一个季度中得分的任务。我们的主要发现有两个方面。首先，我们发现在我们使用的所有模型中，GPT-4 是最有效的，其次是 Claude-2.1，而 GPT-3.5、Gemini-Pro 和 Llama-2-70b 则落后。具体而言，我们比较了三种不同的提示技术和一种分而治之的方法，我们发现后者最有效。我们的分而治之方法将逐个拆解比赛数据为更小且更易处理的片段，分别解决每个片段，然后将它们聚合在一起。除了分而治之的方法，我们还探索了思维链（CoT）策略，该策略明显改善了某些模型，尤其是 GPT-4 和 Claude-2.1 的效果，它们的准确率显著提高。然而，对于 GPT-3.5 和 Gemini-Pro 等其他模型，思维链策略则几乎没有或甚至具有负面影响。其次，令我们惊讶的是，尽管在计算 NFL 季度得分方面表现出色，但大多数模型，包括 GPT-4，却在准确计算 NBA 季度总分方面遇到困难。这促使我们进一步研究影响分析推理任务复杂性的因素，并通过大量实验得出结论，即任务的复杂性取决于上下文长度、信息密度和相关信息的存在。我们的研究为分析推理任务的复杂性和未来大型语言模型的发展方向提供了有价值的见解。