Neural language models have become powerful tools for learning complex
representations of entities in natural language processing tasks. However,
their interpretability remains a significant challenge, particularly in domains
like computational biology where trust in model predictions is crucial. In this
work, we aim to enhance the interpretability of protein language models,
specifically the state-of-the-art ESM model, by identifying and characterizing
knowledge neurons - components that express understanding of key information.
After fine-tuning the ESM model for the task of enzyme sequence classification,
we compare two knowledge neuron selection methods that preserve a subset of
neurons from the original model. The two methods, activation-based and
integrated gradient-based selection, consistently outperform a random baseline.
In particular, these methods show that there is a high density of knowledge
neurons in the key vector prediction networks of self-attention modules. Given
that key vectors specialize in understanding different features of input
sequences, these knowledge neurons could capture knowledge of different enzyme
sequence motifs. In the future, the types of knowledge captured by each neuron
could be characterized.

神经语言模型在自然语言处理任务中学习复杂实体表示成为强大工具，然而在计算生物学等领域中，其可解释性仍然是一个重要挑战。本研究旨在通过识别和表征知识神经元增强蛋白质语言模型（特别是 ESM 模型）的可解释性，通过将 ESM 模型针对酶序列分类任务进行微调，在比较两种保留原始模型中的子集神经元的知识神经元选择方法方面取得了一致优于随机基准线的结果。尤其是这些方法表明自注意力模块的关键向量预测网络中存在高密度的知识神经元，考虑到关键向量专门用于理解输入序列的不同特征，这些知识神经元可以捕捉不同的酶序列基序知识。未来，可以对每个神经元所捕捉到的知识类型进行表征。