We are beginning to see progress in language model assisted scientific
discovery. Motivated by the use of LLMs as a general scientific assistant, this
paper assesses the domain knowledge of LLMs through its understanding of
different mathematical skills required to solve problems. In particular, we
look at not just what the pre-trained model already knows, but how it learned
to learn from information during in-context learning or instruction-tuning
through exploiting the complex knowledge structure within mathematics.
Motivated by the Neural Tangent Kernel (NTK), we propose \textit{NTKEval} to
assess changes in LLM's probability distribution via training on different
kinds of math data. Our systematic analysis finds evidence of domain
understanding during in-context learning. By contrast, certain
instruction-tuning leads to similar performance changes irrespective of
training on different data, suggesting a lack of domain understanding across
different skills.

通过评估预训练语言模型对解决问题所需的不同数学技能的领域知识，本文提出了 NTKEval 方法来评估 LLM 概率分布变化的培训，发现当场景中学到的培训以及利用数学知识结构时存在领域理解。相比之下，某些指令调整导致类似的性能变化，无论培训数据不同，暗示了跨不同技能的领域理解缺乏。

超越模式匹配学习？对 LLM 中的数学理解进行评估

Learning Beyond Pattern Matching? Assaying Mathematical Understanding in  LLMs

Acquiring mathematical skills is considered a key challenge for modern
Artificial Intelligence systems. Inspired by the way humans discover numerical
knowledge, here we introduce a deep reinforcement learning framework that
allows to simulate how cognitive agents could gradually learn to solve
arithmetic problems by interacting with a virtual abacus. The proposed model
successfully learn to perform multi-digit additions and subtractions, achieving
an error rate below 1% even when operands are much longer than those observed
during training. We also compare the performance of learning agents receiving a
different amount of explicit supervision, and we analyze the most common error
patterns to better understand the limitations and biases resulting from our
design choices.

通过深度强化学习框架，模拟认知智能体通过与虚拟珠算交互逐渐学习解决数学问题，并比较不同程度监督下的学习代理的性能及分析设计选择所导致的限制和偏见，研究人员成功地让模型学会了多位数加减，错误率低于 1%。