Parameter-efficient finetuning (PEFT) has become ubiquitous to adapt
foundation models to downstream task requirements while retaining their
generalization ability. However, the amount of additionally introduced
parameters and compute for successful adaptation and hyperparameter searches
can explode quickly, especially when deployed at scale to serve numerous
individual requests. To ensure effective, parameter-efficient, and
hyperparameter-robust adaptation, we propose the ETHER transformation family,
which performs Efficient fineTuning via HypErplane Reflections. By design,
ETHER transformations require a minimal number of parameters, are less likely
to deteriorate model performance, and exhibit robustness to hyperparameter and
learning rate choices. In particular, we introduce ETHER and its relaxation
ETHER+, which match or outperform existing PEFT methods with significantly
fewer parameters ($\sim$$10$-$100$ times lower than LoRA or OFT) across
multiple image synthesis and natural language tasks without exhaustive
hyperparameter tuning. Finally, we investigate the recent emphasis on
Hyperspherical Energy retention for adaptation and raise questions on its
practical utility. The code is available at this https URL

我们提出了 ETHER 转换家族，通过超平面反射来实现高效微调，其需要较少的参数，对超参数和学习率选择具有鲁棒性，并在多个图像合成和自然语言任务中与现有的参数高效微调方法相匹配或超越，同时引发对 Hyperspherical 能量保留在实际上的可用性问题。