While protein language models (pLMs) have transformed biological research,
the scaling laws governing their improvement remain underexplored. By adapting
methodologies from NLP scaling laws, we investigated the optimal ratio between
model parameters and training tokens within a fixed compute budget. Our study
reveals that pLM sizes scale sublinearly with compute budget, showing
diminishing returns in performance as model size increases, and we identify a
performance plateau in training loss comparable to the one found in relevant
works in the field. Our findings suggest that widely-used pLMs might not be
compute-optimal, indicating that larger models could achieve convergence more
efficiently. Training a 35M model on a reduced token set, we attained
perplexity results comparable to larger models like ESM-2 (15B) and xTrimoPGLM
(100B) with a single dataset pass. This work paves the way towards more
compute-efficient pLMs, democratizing their training and practical application
in computational biology.

通过研究蛋白质语言模型的规模定律，我们发现 pLM 的大小在计算预算范围内呈次线性增长，表现出随着模型规模增加性能递减的现象。研究结果显示 pLM 的训练损失也存在性能平稳期，这表明广泛使用的 pLM 可能不会在计算效率上达到最佳状态，而使用更大的模型能更高效地实现收敛。通过在较小的数据集上训练一个 35M 模型，我们得到了与更大模型（如 ESM-2（15B）和 xTrimoPGLM（100B））相当的困惑度结果，这为更高效地训练和在计算生物学中的实际应用 pLM 铺平了道路。