This paper presents ProFL, a novel progressive FL framework to effectively
break the memory wall. Specifically, ProFL divides the model into different
blocks based on its original architecture. Instead of updating the full model
in each training round, ProFL first trains the front blocks and safely freezes
them after convergence. Training of the next block is then triggered. This
process iterates until the training of the whole model is completed. In this
way, the memory footprint is effectively reduced for feasible deployment on
heterogeneous devices. In order to preserve the feature representation of each
block, we decouple the whole training process into two stages: progressive
model shrinking and progressive model growing. During the progressive model
shrinking stage, we meticulously design corresponding output modules to assist
each block in learning the expected feature representation and obtain the
initialization parameters. Then, the obtained output modules are utilized in
the corresponding progressive model growing stage. Additionally, to control the
training pace for each block, a novel metric from the scalar perspective is
proposed to assess the learning status of each block and determines when to
trigger the training of the next one. Finally, we theoretically prove the
convergence of ProFL and conduct extensive experiments on representative models
and datasets to evaluate the effectiveness of ProFL. The results demonstrate
that ProFL effectively reduces the peak memory footprint by up to 57.4% and
improves model accuracy by up to 82.4%.

通过分割模型结构，分阶段训练并冻结模型的不同部分，逐步减小模型的尺寸以降低内存占用，从而提高性能。