Proteins are essential to life's processes, underpinning evolution and
diversity. Advances in sequencing technology have revealed millions of
proteins, underscoring the need for sophisticated pre-trained protein models
for biological analysis and AI development. Facebook's ESM2, the most advanced
protein language model to date, leverages a masked prediction task for
unsupervised learning, crafting amino acid representations with notable
biochemical accuracy. Yet, it lacks in delivering functional protein insights,
signaling an opportunity for enhancing representation quality.Our study
addresses this gap by incorporating protein family classification into ESM2's
training.This approach, augmented with Community Propagation-Based Clustering
Algorithm, improves global protein representations, while a contextual
prediction task fine-tunes local amino acid accuracy. Significantly, our model
achieved state-of-the-art results in several downstream experiments,
demonstrating the power of combining global and local methodologies to
substantially boost protein representation quality.

本研究使用 ESM2 最先进的蛋白质语言模型，并结合蛋白质家族分类和基于社区传播的聚类算法，改善了全局蛋白质表示和局部氨基酸准确性，从而大幅提升蛋白质表达质量，并在多个下游实验中取得了最先进的结果。

超越 ESM2: 借助高效聚类的图增强蛋白质序列建模

Beyond ESM2: Graph-Enhanced Protein Sequence Modeling with Efficient  Clustering

Pre-trained protein models (PTPMs) represent a protein with one fixed
embedding and thus are not capable for diverse tasks. For example, protein
structures can shift, namely protein folding, between several conformations in
various biological processes. To enable PTPMs to produce task-aware
representations, we propose to learn interpretable, pluggable and extensible
protein prompts as a way of injecting task-related knowledge into PTPMs. In
this regard, prior PTPM optimization with the masked language modeling task can
be interpreted as learning a sequence prompt (Seq prompt) that enables PTPMs to
capture the sequential dependency between amino acids. To incorporate
conformational knowledge to PTPMs, we propose an interaction-conformation
prompt (IC prompt) that is learned through back-propagation with the
protein-protein interaction task. As an instantiation, we present a
conformation-aware pre-trained protein model that learns both sequence and
interaction-conformation prompts in a multi-task setting. We conduct
comprehensive experiments on nine protein datasets. Results confirm our
expectation that using the sequence prompt does not hurt PTPMs' performance on
sequence-related tasks while incorporating the interaction-conformation prompt
significantly improves PTPMs' performance on tasks where conformational
knowledge counts. We also show the learned prompts can be combined and extended
to deal with new complex tasks.

通过学习可解释、可插入、可扩展的蛋白质提示，将任务相关知识注入预训练蛋白质模型（PTPMs），从而使得 PTPMs 能够生成任务感知的表示，进而在多任务设置下，学习同时序列与交互构象提示的构象感知预训练蛋白质模型，实验证明这种方法显著提高了 PTPMs 在需要构象知识的任务上的性能。