The existing methods for evaluating the inference abilities of Large Language
Models (LLMs) have been results-centric, making it difficult to assess the
inference process. We introduce a new approach using the Abstract and Reasoning
Corpus (ARC) dataset to evaluate the inference and contextual understanding
abilities of large language models in a process-centric manner. ARC demands
rigorous logical structures for problem-solving, making it a benchmark that
facilitates the comparison of model inference abilities with humans.
Experimental results confirm that while large language models possess weak
inference abilities, they still lag in terms of logical coherence,
compositionality, and productivity. Our experiments highlight the reasoning
capabilities of LLMs, proposing development paths for achieving human-level
reasoning.

利用 Abstract and Reasoning Corpus 数据集的新方法，我们评估了大型语言模型的推理和上下文理解能力，结果表明虽然大型语言模型具有薄弱的推理能力，但在逻辑连贯性、组合性和生产力方面仍然落后于人类，我们的实验突显了 LLM 的推理能力，并提出了实现人类级别推理的发展路径。