This paper aims to prove the emergence of symbolic concepts in well-trained
AI models. We prove that if (1) the high-order derivatives of the model output
w.r.t. the input variables are all zero, (2) the AI model can be used on
occluded samples and will yield higher confidence when the input sample is less
occluded, and (3) the confidence of the AI model does not significantly degrade
on occluded samples, then the AI model will encode sparse interactive concepts.
Each interactive concept represents an interaction between a specific set of
input variables, and has a certain numerical effect on the inference score of
the model. Specifically, it is proved that the inference score of the model can
always be represented as the sum of the interaction effects of all interactive
concepts. In fact, we hope to prove that conditions for the emergence of
symbolic concepts are quite common. It means that for most AI models, we can
usually use a small number of interactive concepts to mimic the model outputs
on any arbitrarily masked samples.

通过证明 AI 模型对不同交互概念的输入输出方式的映射，本研究试图证明，大多数 AI 模型可使用少数交互概念来在任意遮掩样本上模拟模型输出。

AI 模型中稀疏符号概念出现证明的进展

Where We Have Arrived in Proving the Emergence of Sparse Symbolic  Concepts in AI Models

Many complex problems are naturally understood in terms of symbolic concepts.
For example, our concept of "cat" is related to our concepts of "ears" and
"whiskers" in a non-arbitrary way. Fodor (1998) proposes one theory of
concepts, which emphasizes symbolic representations related via constituency
structures. Whether neural networks are consistent with such a theory is open
for debate. We propose unit tests for evaluating whether a system's behavior is
consistent with several key aspects of Fodor's criteria. Using a simple visual
concept learning task, we evaluate several modern neural architectures against
this specification. We find that models succeed on tests of groundedness,
modularlity, and reusability of concepts, but that important questions about
causality remain open. Resolving these will require new methods for analyzing
models' internal states.

本研究提出了关于符号概念的理论，并使用一个简单的视觉概念学习任务来评估多个现代神经结构与该理论的一致程度，发现这些模型可以成功通过测验，但在因果关系方面仍然存在重要的问题。