Fine-tuning Pre-trained protein language models (PLMs) has emerged as a
prominent strategy for enhancing downstream prediction tasks, often
outperforming traditional supervised learning approaches. As a widely applied
powerful technique in natural language processing, employing
Parameter-Efficient Fine-Tuning techniques could potentially enhance the
performance of PLMs. However, the direct transfer to life science tasks is
non-trivial due to the different training strategies and data forms. To address
this gap, we introduce SES-Adapter, a simple, efficient, and scalable adapter
method for enhancing the representation learning of PLMs. SES-Adapter
incorporates PLM embeddings with structural sequence embeddings to create
structure-aware representations. We show that the proposed method is compatible
with different PLM architectures and across diverse tasks. Extensive
evaluations are conducted on 2 types of folding structures with notable quality
differences, 9 state-of-the-art baselines, and 9 benchmark datasets across
distinct downstream tasks. Results show that compared to vanilla PLMs,
SES-Adapter improves downstream task performance by a maximum of 11% and an
average of 3%, with significantly accelerated training speed by a maximum of
1034% and an average of 362%, the convergence rate is also improved by
approximately 2 times. Moreover, positive optimization is observed even with
low-quality predicted structures. The source code for SES-Adapter is available
at this https URL

SES-Adapter 通过将 PLM 嵌入与结构序列嵌入相结合，创建了结构感知的表示方法，进一步提高了下游任务性能。与普通的 PLMs 相比，SES-Adapter 最大程度上提高了下游任务性能达 11%，平均提高了 3%，训练速度也加快了最高 1034% 和平均 362%。即使在低质量的预测结构下，也观察到正向优化。