Attention-based models trained on protein sequences have demonstrated
incredible success at classification and generation tasks relevant for
artificial intelligence-driven protein design. However, we lack a sufficient
understanding of how very large-scale models and data play a role in effective
protein model development. We introduce a suite of protein language models,
named ProGen2, that are scaled up to 6.4B parameters and trained on different
sequence datasets drawn from over a billion proteins from genomic, metagenomic,
and immune repertoire databases. ProGen2 models show state-of-the-art
performance in capturing the distribution of observed evolutionary sequences,
generating novel viable sequences, and predicting protein fitness without
additional finetuning. As large model sizes and raw numbers of protein
sequences continue to become more widely accessible, our results suggest that a
growing emphasis needs to be placed on the data distribution provided to a
protein sequence model. We release the ProGen2 models and code at
this https URL

介绍了一套名为 ProGen2 的蛋白质语言模型，它具有 64 亿个参数，在来自多种数据库的超过 10 亿个蛋白质序列数据集上进行训练。ProGen2 模型表现出捕获已观察到的进化序列分布，生成新的可行序列以及在不需要额外微调的情况下预测蛋白质适应性的最佳性能。研究表明在向蛋白质序列模型提供数据分布时需要越来越多的关注点。