This paper investigates the inherent knowledge in language models from the
perspective of epistemological holism. The purpose of this paper is to explore
whether LLMs exhibit characteristics consistent with epistemological holism.
These characteristics suggest that core knowledge, such as general scientific
knowledge, each plays a specific role, serving as the foundation of our
knowledge system and being difficult to revise. To assess these traits related
to holism, we created a scientific reasoning dataset and examined the
epistemology of language models through three tasks: Abduction, Revision, and
Argument Generation. In the abduction task, the language models explained
situations while avoiding revising the core knowledge. However, in other tasks,
the language models were revealed not to distinguish between core and
peripheral knowledge, showing an incomplete alignment with holistic knowledge
principles.

研究以认识论整体主义的观点调查语言模型中的固有知识，通过构建科学推理数据集并进行推理、修正和论证生成三个任务评估语言模型的认识论。结果显示，在问题解释阶段，语言模型可以解释情境而避免修正核心知识，然而在其他任务中，语言模型则未能区分核心知识和周边知识，显示对认识论整体主义原则的不完全一致性。

语言模型的认识论：语言模型是否拥有整体性知识？

Epistemology of Language Models: Do Language Models Have Holistic  Knowledge?

Can a Large Language Model (LLM) solve simple abstract reasoning problems? We
explore this broad question through a systematic analysis of GPT on the
Abstraction and Reasoning Corpus (ARC), a representative benchmark of abstract
reasoning ability from limited examples in which solutions require some "core
knowledge" of concepts such as objects, goal states, counting, and basic
geometry. GPT-4 solves only 13/50 of the most straightforward ARC tasks when
using textual encodings for their two-dimensional input-output grids. Our
failure analysis reveals that GPT-4's capacity to identify objects and reason
about them is significantly influenced by the sequential nature of the text
that represents an object within a text encoding of a task. To test this
hypothesis, we design a new benchmark, the 1D-ARC, which consists of
one-dimensional (array-like) tasks that are more conducive to GPT-based
reasoning, and where it indeed performs better than on the (2D) ARC. To
alleviate this issue, we propose an object-based representation that is
obtained through an external tool, resulting in nearly doubling the performance
on solved ARC tasks and near-perfect scores on the easier 1D-ARC. Although the
state-of-the-art GPT-4 is unable to "reason" perfectly within non-language
domains such as the 1D-ARC or a simple ARC subset, our study reveals that the
use of object-based representations can significantly improve its reasoning
ability. Visualizations, GPT logs, and data are available at
this https URL

本文探讨了大型语言模型在抽象推理问题中的解决能力，提出了基于物体表示的解决方案，解决了 ARC 数据集中的 13 个问题，并在 1D-ARC 数据集上取得了接近完美的成绩。

LLMs 与抽象推理数据集：成功、失败及基于对象表示的重要性

LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and  the Importance of Object-based Representations

A large literature suggests that people are intuitive Dualists--they consider
the mind ethereal, distinct from the body. Past research also shows that
Dualism emerges, in part, via learning (e.g., Barlev & Shtulman, 2021). But
whether learning is sufficient to give rise to Dualism is unknown.The evidence
from human learners does address this question because humans are endowed not
only with general learning capacities but also with core knowledge capacities.
And recent results suggest that core knowledge begets Dualism (Berent, Theodore
& Valencia, 2021; Berent, 2023). To evaluate the role of learning, here, we
probe for a mind-body divide in Davinci--a large language model (LLM) that is
devoid of any innate core knowledge. We show that Davinci still leans towards
Dualism, and that this bias increases systematically with the learner's
inductive potential. Thus, davinci (a GPT-3 model) exhibits mild Dualist
tendencies, whereas its descendent, text-davinci-003 (a GPT-3.5 model), shows a
full-blown bias. It selectively considers thoughts (epistemic states) as
disembodied--as unlikely to show up in the body (in the brain), but not in its
absence (after death). While Davinci's performance is constrained by its
syntactic limitations, and it differs from humans, its Dualist bias is robust.
These results demonstrate that the mind-body divide is partly learnable from
experience.They also show how, as LLM's are exposed to human narratives, they
induce not only human knowledge but also human biases.

本文利用 Davinci 作为一个大语言模型，探究是否学习足以产生二元论，发现其受推理能力影响呈现二元论倾向，这一结果表明心 - 身分离在一定程度上是可以通过经验学习的，并且大型语言模型在接触人类故事时不仅学习人类知识，也会学习人类偏见。