While large language models have made strides in natural language processing,
their proficiency in complex reasoning tasks requiring formal language
comprehension, such as chess, remains less investigated. This paper probes the
performance of ChatGPT, a sophisticated language model by OpenAI in tackling
such complex reasoning tasks, using chess as a case study. Through robust
metrics examining both the legality and quality of moves, we assess ChatGPT's
understanding of the chessboard, adherence to chess rules, and strategic
decision-making abilities. Our evaluation identifies limitations within
ChatGPT's attention mechanism that affect its formal language comprehension and
uncovers the model's underdeveloped self-regulation abilities. Our study also
reveals ChatGPT's propensity for a coherent strategy in its gameplay and a
noticeable uptick in decision-making assertiveness when the model is presented
with a greater volume of natural language or possesses a more lucid
understanding of the state of the chessboard. These findings contribute to the
growing exploration of language models' abilities beyond natural language
processing, providing valuable information for future research towards models
demonstrating human-like cognitive abilities.

这篇论文通过探究 ChatGPT（OpenAI 的先进语言模型）处理复杂推理任务（以国际象棋为例）的表现，利用评估棋盘理解、国际象棋规则遵守和战略决策能力的健全指标来识别其注意机制和自我调节能力的局限性，以及在自然语言体量较大或棋盘状态理解较明晰时决策力的增强；这些研究结果对于突破自然语言处理的局限，为展示类人认知能力的模型提供了宝贵的信息。