The goal of protein representation learning is to extract knowledge from
protein databases that can be applied to various protein-related downstream
tasks. Although protein sequence, structure, and function are the three key
modalities for a comprehensive understanding of proteins, existing methods for
protein representation learning have utilized only one or two of these
modalities due to the difficulty of capturing the asymmetric interrelationships
between them. To account for this asymmetry, we introduce our novel asymmetric
multi-modal masked autoencoder (AMMA). AMMA adopts (1) a unified multi-modal
encoder to integrate all three modalities into a unified representation space
and (2) asymmetric decoders to ensure that sequence latent features reflect
structural and functional information. The experiments demonstrate that the
proposed AMMA is highly effective in learning protein representations that
exhibit well-aligned inter-modal relationships, which in turn makes it
effective for various downstream protein-related tasks.

蛋白质表示学习旨在从蛋白质数据库中提取知识，以应用于各种蛋白质相关的下游任务，我们引入了新颖的不对称多模态掩码自编码器（AMMA），利用统一多模态编码器将蛋白质序列、结构和功能三种关键形态集成为统一的表示空间，通过不对称解码器确保序列潜在特征反映结构和功能信息，实验表明 AMMA 在学习展现良好间模态关系的蛋白质表示方面具有极高的效果，进而使其在各种蛋白质相关的下游任务中发挥作用。

捕捉蛋白质序列 - 结构 - 功能关系的蛋白质表示学习

Protein Representation Learning by Capturing Protein  Sequence-Structure-Function Relationship

The prediction of protein structures from sequences is an important task for
function prediction, drug design, and related biological processes
understanding. Recent advances have proved the power of language models (LMs)
in processing the protein sequence databases, which inherit the advantages of
attention networks and capture useful information in learning representations
for proteins. The past two years have witnessed remarkable success in tertiary
protein structure prediction (PSP), including evolution-based and
single-sequence-based PSP. It seems that instead of using energy-based models
and sampling procedures, protein language model (pLM)-based pipelines have
emerged as mainstream paradigms in PSP. Despite the fruitful progress, the PSP
community needs a systematic and up-to-date survey to help bridge the gap
between LMs in the natural language processing (NLP) and PSP domains and
introduce their methodologies, advancements and practical applications. To this
end, in this paper, we first introduce the similarities between protein and
human languages that allow LMs extended to pLMs, and applied to protein
databases. Then, we systematically review recent advances in LMs and pLMs from
the perspectives of network architectures, pre-training strategies,
applications, and commonly-used protein databases. Next, different types of
methods for PSP are discussed, particularly how the pLM-based architectures
function in the process of protein folding. Finally, we identify challenges
faced by the PSP community and foresee promising research directions along with
the advances of pLMs. This survey aims to be a hands-on guide for researchers
to understand PSP methods, develop pLMs and tackle challenging problems in this
field for practical purposes.

本文系统地概述了蛋白质语言模型在蛋白质结构预测中的应用和方法，介绍了网络架构、预训练策略、常用蛋白质数据库等方面的最新进展和挑战，并展望了未来发展方向。