Generative protein language models are a natural way to design new proteins
with desired functions. However, current models are either difficult to direct
to produce a protein from a specific family of interest, or must be trained on
a large multiple sequence alignment (MSA) from the specific family of interest,
making them unable to benefit from transfer learning across families. To
address this, we propose $\textbf{P}$r$\textbf{o}$tein $\textbf{E}$volutionary
$\textbf{T}$ransformer (PoET), an autoregressive generative model of whole
protein families that learns to generate sets of related proteins as
sequences-of-sequences across tens of millions of natural protein sequence
clusters. PoET can be used as a retrieval-augmented language model to generate
and score arbitrary modifications conditioned on any protein family of
interest, and can extrapolate from short context lengths to generalize well
even for small families. This is enabled by a unique Transformer layer; we
model tokens sequentially within sequences while attending between sequences
order invariantly, allowing PoET to scale to context lengths beyond those used
during training. PoET outperforms existing protein language models and
evolutionary sequence models for variant function prediction in extensive
experiments on deep mutational scanning datasets, improving variant effect
prediction across proteins of all MSA depths.

提出了一种全新的蛋白质进化转换器（PoET）, 它是一种自回归生成整个蛋白家族的模型，可以用作检索嵌入式语言模型，从而生成和评分任意修饰物，并对深度突变扫描数据集进行广泛实验，可以改善所有 MSA 深度的蛋白质变体效应预测。