As model sizes grow, finding efficient and cost-effective hyperparameter optimization (HPO) methods becomes increasingly crucial for deep learning pipelines. While multi-fidelity HPO (MF-HPO) trades off computational resources required for DL training with lower fidelity estimations, existing fidelity sources often fail under lower compute and memory constraints. We propose a novel fidelity source: the number of layers that are trained or frozen during training. For deep networks, this approach offers significant compute and memory savings while preserving rank correlations between hyperparameters at low fidelities compared to full model training. We demonstrate this in our empirical evaluation across ResNets and Transformers and additionally analyze the utility of frozen layers as a fidelity in using GPU resources as a fidelity in HPO, and for a combined MF-HPO with other fidelity sources. This contribution opens new applications for MF-HPO with hardware resources as a fidelity and creates opportunities for improved algorithms navigating joint fidelity spaces.

本研究解决了随着模型规模增长，深度学习管道中高效、经济的超参数优化方法的需求问题。提出了一种新颖的方法，即在训练过程中冻结或训练的层数作为保真度源，这种方法在计算和内存方面提供了显著节省，同时保持了低保真度下超参数之间的秩相关性。实证评估表明，该贡献为使用硬件资源作为保真度的多保真度超参数优化开辟了新应用，并为改进算法在联合保真空间中的导航提供了机会。

冻结层：节省内存的多保真度超参数优化