Large language models (LLMs) have developed impressive performance and strong
explainability across various reasoning scenarios, marking a significant stride
towards mimicking human-like intelligence. Despite this, when tasked with
simple questions supported by a generic fact, LLMs often fail to provide
consistent and precise answers, indicating a deficiency in abstract reasoning
abilities. This has sparked a vigorous debate about whether LLMs are genuinely
reasoning or merely memorizing. In light of this, we design a preliminary study
to quantify and delve into the abstract reasoning abilities of existing LLMs.
Our findings reveal a substantial discrepancy between their general reasoning
and abstract reasoning performances. To relieve this problem, we tailor an
abstract reasoning dataset (AbsR) together with a meaningful learning paradigm
to teach LLMs how to leverage generic facts for reasoning purposes. The results
show that our approach not only boosts the general reasoning performance of
LLMs but also makes considerable strides towards their capacity for abstract
reasoning, moving beyond simple memorization or imitation to a more nuanced
understanding and application of generic facts.

我们设计了一个初步的研究，以量化和深入探讨现有大型语言模型的抽象推理能力。我们的结果显示，我们的方法不仅提高了大型语言模型的一般推理性能，而且在抽象推理能力方面取得了可观的进展，从简单的记忆或模仿转向了更加精细的对通用事实的理解和应用。

有意义的学习：通过通用事实指导提升大型语言模型的抽象推理能力

Meaningful Learning: Advancing Abstract Reasoning in Large Language  Models via Generic Fact Guidance

We explore the abstract reasoning abilities of text-only and multimodal
versions of GPT-4, using the ConceptARC benchmark [10], which is designed to
evaluate robust understanding and reasoning with core-knowledge concepts. We
extend the work of Moskvichev et al. [10] by evaluating GPT-4 on more detailed,
one-shot prompting (rather than simple, zero-shot prompts) with text versions
of ConceptARC tasks, and by evaluating GPT-4V, the multimodal version of GPT-4,
on zero- and one-shot prompts using image versions of the simplest tasks. Our
experimental results support the conclusion that neither version of GPT-4 has
developed robust abstraction abilities at humanlike levels.

我们探索了 GPT-4 的纯文本和多模态版本的抽象推理能力，使用 ConceptARC 基准测试评估核心概念的严谨理解和推理能力。我们通过在更详细的一次性提示上评估纯文本版本的 ConceptARC 任务（而不是简单的零次提示），以及通过使用最简单任务的图像版本在零次和一次提示上评估 GPT-4 的多模态版本 GPT-4V，扩展了 Moskvichev 等人的工作。我们的实验结果支持结论：GPT-4 的任何版本都没有达到人类水平的严谨抽象能力。