AlphaFold2 has been hailed as a breakthrough in protein folding. It can rapidly predict protein structures with lab-grade accuracy. However, its implementation does not include the necessary training code. OpenFold is the first trainable public reimplementation of AlphaFold. AlphaFold training procedure is prohibitively time-consuming, and gets diminishing benefits from scaling to more compute resources. In this work, we conducted a comprehensive analysis on the AlphaFold training procedure based on Openfold, identified that inefficient communications and overhead-dominated computations were the key factors that prevented the AlphaFold training from effective scaling. We introduced ScaleFold, a systematic training method that incorporated optimizations specifically for these factors. ScaleFold successfully scaled the AlphaFold training to 2080 NVIDIA H100 GPUs with high resource utilization. In the MLPerf HPC v3.0 benchmark, ScaleFold finished the OpenFold benchmark in 7.51 minutes, shown over $6\times$ speedup than the baseline. For training the AlphaFold model from scratch, ScaleFold completed the pretraining in 10 hours, a significant improvement over the seven days required by the original AlphaFold pretraining baseline.

AlphaFold2是蛋白质折叠领域的突破性成果，但其实现不包括必要的训练代码。OpenFold是AlphaFold的第一个可训练的公共重新实现。本研究在OpenFold的基础上对AlphaFold的训练过程进行了全面分析，发现低效的通信和资源开销过大的计算是导致AlphaFold训练无法有效扩展的关键因素。我们提出了ScaleFold，这是一种系统化的训练方法，专门针对这些因素进行了优化。ScaleFold成功将AlphaFold训练扩展到2080个NVIDIA H100 GPU，资源利用率高。在MLPerf HPC v3.0基准测试中，ScaleFold在7.51分钟内完成了OpenFold基准测试，比基线快了6倍。对于从头开始训练AlphaFold模型，ScaleFold的预训练只需10小时，比原始AlphaFold预训练基线的七天显著提高。

ScaleFold: 将AlphaFold初始训练时间缩短至10小时