Reasoning is most powerful when an LLM accurately aggregates relevant
information. We examine the critical role of information aggregation in
reasoning by requiring the LLM to analyze sports narratives. To succeed at this
task, an LLM must infer points from actions, identify related entities,
attribute points accurately to players and teams, and compile key statistics to
draw conclusions. We conduct comprehensive experiments with real NBA basketball
data and present SportsGen, a new method to synthesize game narratives. By
synthesizing data, we can rigorously evaluate LLMs' reasoning capabilities
under complex scenarios with varying narrative lengths and density of
information. Our findings show that most models, including GPT-4o, often fail
to accurately aggregate basketball scores due to frequent scoring patterns.
Open-source models like Llama-3 further suffer from significant score
hallucinations. Finally, the effectiveness of reasoning is influenced by
narrative complexity, information density, and domain-specific terms,
highlighting the challenges in analytical reasoning tasks.

综合真实 NBA 篮球数据，我们提出了一种名为 SportsGen 的新方法，用于综合比赛叙述，以严谨评估复杂情景下 LLMs 的推理能力，结果表明 GPT-4o 等大多数模型常因频繁的得分模式而不能准确汇总篮球比分，而开源模型如 Llama-3 则存在显著的得分幻觉，最后我们发现推理的效果受叙述复杂性、信息密度和领域特定术语的影响，突显了分析推理任务的挑战。