We are beginning to see progress in language model assisted scientific
discovery. Motivated by the use of LLMs as a general scientific assistant, this
paper assesses the domain knowledge of LLMs through its understanding of
different mathematical skills required to solve problems. In particular, we
look at not just what the pre-trained model already knows, but how it learned
to learn from information during in-context learning or instruction-tuning
through exploiting the complex knowledge structure within mathematics.
Motivated by the Neural Tangent Kernel (NTK), we propose \textit{NTKEval} to
assess changes in LLM's probability distribution via training on different
kinds of math data. Our systematic analysis finds evidence of domain
understanding during in-context learning. By contrast, certain
instruction-tuning leads to similar performance changes irrespective of
training on different data, suggesting a lack of domain understanding across
different skills.

通过评估预训练语言模型对解决问题所需的不同数学技能的领域知识，本文提出了 NTKEval 方法来评估 LLM 概率分布变化的培训，发现当场景中学到的培训以及利用数学知识结构时存在领域理解。相比之下，某些指令调整导致类似的性能变化，无论培训数据不同，暗示了跨不同技能的领域理解缺乏。