Protein language models leverage evolutionary information to perform
state-of-the-art 3D structure and zero-shot variant prediction. Yet, extracting
and explaining all the mutational interactions that govern model predictions
remains difficult as it requires querying the entire amino acid space for $n$
sites using $20^n$ sequences, which is computationally expensive even for
moderate values of $n$ (e.g., $n\sim10$). Although approaches to lower the
sample complexity exist, they often limit the interpretability of the model to
just single and pairwise interactions. Recently, computationally scalable
algorithms relying on the assumption of sparsity in the Fourier domain have
emerged to learn interactions from experimental data. However, extracting
interactions from language models poses unique challenges: it's unclear if
sparsity is always present or if it is the only metric needed to assess the
utility of Fourier algorithms. Herein, we develop a framework to do a
systematic Fourier analysis of the protein language model ESM2 applied on three
proteins-green fluorescent protein (GFP), tumor protein P53 (TP53), and G
domain B1 (GB1)-across various sites for 228 experiments. We demonstrate that
ESM2 is dominated by three regions in the sparsity-ruggedness plane, two of
which are better suited for sparse Fourier transforms. Validations on two
sample proteins demonstrate recovery of all interactions with $R^2=0.72$ in the
more sparse region and $R^2=0.66$ in the more dense region, using only 7
million out of $20^{10}\sim10^{13}$ ESM2 samples, reducing the computational
time by a staggering factor of 15,000. All codes and data are available on our
GitHub repository this https URL

基于 ESM2 的蛋白质语言模型的傅里叶分析发现了在稀疏 - 崎岖平面上占据主导地位的三个区域，其中两个区域更适合进行稀疏傅里叶变换，通过使用仅 700 万个 ESM2 样本中的相互作用可以在更稀疏的区域获得 0.72 的 R^2，而在更密集的区域可以获得 0.66 的 R^2，将计算时间减少了一万五千倍。

从蛋白质语言模型中恢复高阶相互作用

On Recovering Higher-order Interactions from Protein Language Models

Predicting protein properties is paramount for biological and medical
advancements. Current protein engineering mutates on a typical protein, called
the wild-type, to construct a family of homologous proteins and study their
properties. Yet, existing methods easily neglect subtle mutations, failing to
capture the effect on the protein properties. To this end, we propose EvolMPNN,
Evolution-aware Message Passing Neural Network, to learn evolution-aware
protein embeddings. EvolMPNN samples sets of anchor proteins, computes
evolutionary information by means of residues and employs a differentiable
evolution-aware aggregation scheme over these sampled anchors. This way
EvolMPNNcan capture the mutation effect on proteins with respect to the anchor
proteins. Afterwards, the aggregated evolution-aware embeddings are integrated
with sequence embeddings to generate final comprehensive protein embeddings.
Our model shows up to 6.4% better than state-of-the-art methods and attains 36x
inference speedup in comparison with large pre-trained models.

为了预测蛋白质属性，我们提出了 EvolMPNN，即进化感知的消息传递神经网络，通过采样锚定蛋白质集合、计算进化信息以及不可微分的进化感知聚合方案，学习进化感知的蛋白质嵌入，并将其与序列嵌入相结合，生成综合蛋白质嵌入，该模型在性能和推理速度方面优于现有方法。

EvolMPNN: 通过进化编码预测同源蛋白质的突变效应

EvolMPNN: Predicting Mutational Effect on Homologous Proteins by  Evolution Encoding

Protein is linked to almost every life process. Therefore, analyzing the
biological structure and property of protein sequences is critical to the
exploration of life, as well as disease detection and drug discovery.
Traditional protein analysis methods tend to be labor-intensive and
time-consuming. The emergence of deep learning models makes modeling data
patterns in large quantities of data possible. Interdisciplinary researchers
have begun to leverage deep learning methods to model large biological
datasets, e.g. using long short-term memory and convolutional neural network
for protein sequence classification. After millions of years of evolution,
evolutionary information is encoded in protein sequences. Inspired by the
similarity between natural language and protein sequences, we use large-scale
language models to model evolutionary-scale protein sequences, encoding protein
biology information in representation. Significant improvements are observed in
both token-level and sequence-level tasks, demonstrating that our large-scale
model can accurately capture evolution information from pretraining on
evolutionary-scale individual sequences. Our code and model are available at
this https URL

使用大规模的语言模型，可以从蛋白质序列中准确捕捉到进化信息，并在标记层和序列层任务中取得了显着的改进。

利用大规模预训练语言模型建模蛋白质

Modeling Protein Using Large-scale Pretrain Language Model

Deep learning is catalyzing a scientific revolution fueled by big data,
accessible toolkits, and powerful computational resources, impacting many
fields including protein structural modeling. Protein structural modeling, such
as predicting structure from amino acid sequence and evolutionary information,
designing proteins toward desirable functionality, or predicting properties or
behavior of a protein, is critical to understand and engineer biological
systems at the molecular level. In this review, we summarize the recent
advances in applying deep learning techniques to tackle problems in protein
structural modeling and design. We dissect the emerging approaches using deep
learning techniques for protein structural modeling, and discuss advances and
challenges that must be addressed. We argue for the central importance of
structure, following the "sequence -> structure -> function" paradigm. This
review is directed to help both computational biologists to gain familiarity
with the deep learning methods applied in protein modeling, and computer
scientists to gain perspective on the biologically meaningful problems that may
benefit from deep learning techniques.

该综述总结了近年来将深度学习技术应用于蛋白质结构建模及设计问题的最新进展，讨论了已经出现和需解决的挑战，针对该领域提出了许多有益的见解。