Large Language Models have shown tremendous performance on a large variety of
natural language processing tasks, ranging from text comprehension to common
sense reasoning. However, the mechanisms responsible for this success remain
unknown, and it is unclear whether LLMs can achieve human-like cognitive
capabilities or whether these models are still fundamentally limited. Abstract
reasoning is a fundamental task for cognition, consisting of finding and
applying a general pattern from few data. Evaluating deep neural architectures
on this task could give insight into their potential limitations regarding
reasoning and their broad generalisation abilities, yet this is currently an
under-explored area. In this paper, we perform extensive evaluations of
state-of-the-art LLMs on abstract reasoning tasks, showing that they achieve
very limited performance in contrast with other natural language tasks, and we
investigate the reasons for this difference. We apply techniques that have been
shown to improve performance on other NLP tasks and show that in most cases
their impact on abstract reasoning performance is limited. In the course of
this work, we have generated a new benchmark for evaluating language models on
abstract reasoning tasks.

本论文评估了最新的 Large Language Models 在抽象推理任务上的表现，并发现它们相比于其他自然语言处理任务的表现非常有限。作者探讨了这种差异的原因，并提出了一个新的基准，用于评估自然语言处理中的抽象推理任务。