The New York Times Connections game has emerged as a popular and challenging
pursuit for word puzzle enthusiasts. We collect 200 Connections games to
evaluate the performance of state-of-the-art large language models (LLMs)
against expert and novice human players. Our results show that even the
best-performing LLM, GPT-4o, which has otherwise shown impressive reasoning
abilities on a wide variety of benchmarks, can only fully solve 8% of the
games. Compared to GPT-4o, novice and expert players perform better, with
expert human players significantly outperforming GPT-4o. To deepen our
understanding we create a taxonomy of the knowledge types required to
successfully categorize words in the Connections game, revealing that LLMs
struggle with associative, encyclopedic, and linguistic knowledge. Our findings
establish the New York Times Connections game as a challenging benchmark for
evaluating abstract reasoning capabilities in humans and AI systems.

《纽约时报连线游戏对大型语言模型的性能评估》揭示了大型语言模型在解决连线游戏时的局限性，同时发现专家玩家在这方面表现更好，为人工智能系统的抽象推理能力提供了具有挑战性的评估基准。