Background: Deep neural networks have proven to be powerful computational
tools for modeling, prediction, and generation. However, the workings of these
models have generally been opaque. Recent work has shown that the performance
of some models are modulated by overlapping functional networks of connections
within the models. Here the techniques of functional neuroimaging are applied
to an exemplary large language model to probe its functional structure.
Methods: A series of block-designed task-based prompt sequences were generated
to probe the Facebook Galactica-125M model. Tasks included prompts relating to
political science, medical imaging, paleontology, archeology, pathology, and
random strings presented in an off/on/off pattern with prompts about other
random topics. For the generation of each output token, all layer output values
were saved to create an effective time series. General linear models were fit
to the data to identify layer output values which were active with the tasks.
Results: Distinct, overlapping networks were identified with each task. Most
overlap was observed between medical imaging and pathology networks. These
networks were repeatable across repeated performance of related tasks, and
correspondence of identified functional networks and activation in tasks not
used to define the functional networks was shown to accurately identify the
presented task. Conclusion: The techniques of functional neuroimaging can be
applied to deep neural networks as a means to probe their workings. Identified
functional networks hold the potential for use in model alignment, modulation
of model output, and identifying weights to target in fine-tuning.

深度神经网络的功能结构通过功能神经影像学的技术得到了探测。确定的功能网络可以用于模型对齐、模型输出的调节以及在微调中确定权重。

深度神经网络中的功能激活

On Functional Activations in Deep Neural Networks

Long-form answers, consisting of multiple sentences, can provide nuanced and
comprehensive answers to a broader set of questions. To better understand this
complex and understudied task, we study the functional structure of long-form
answers collected from three datasets, ELI5, WebGPT and Natural Questions. Our
main goal is to understand how humans organize information to craft complex
answers. We develop an ontology of six sentence-level functional roles for
long-form answers, and annotate 3.9k sentences in 640 answer paragraphs.
Different answer collection methods manifest in different discourse structures.
We further analyze model-generated answers -- finding that annotators agree
less with each other when annotating model-generated answers compared to
annotating human-written answers. Our annotated data enables training a strong
classifier that can be used for automatic analysis. We hope our work can
inspire future research on discourse-level modeling and evaluation of long-form
QA systems.

本篇论文通过研究三个数据集中的长篇答案，开发了一个六个句子级功能角色的本体论，并进一步分析了模型生成的答案。通过对 3.9k 句子和 640 个答案段落进行注释，我们希望能够鼓励未来的学者进行话语层模型的研究和长篇问答系统的评估。