To address the high computational and parameter complexity associated with
fine-tuning large pre-trained models, researchers have developed
parameter-efficient methods, where only partial parameters are updated for
downstream tasks. However, these works often overlook the distinct properties
of convolutional kernels, which still remain essential elements in many large
models, such as Stable Diffusion. In this study, we first introduce filter
subspace by decomposing convolutional kernels within each network layer over a
small set of filter subspace elements, referred to as filter atoms. We then
fine-tune these models to extract task-specific representation by only adapting
the filter atoms, a few hundred parameters typically. To potentially expand the
parameter space for tuning, we further show a simple approach to generate an
overcomplete filter subspace by recursively decomposing each filter atom over
another set of filter atoms. The fine-tuning of filter atoms reshapes the
filter subspace, enabling convolutional layers to adapt to diverse downstream
tasks efficiently. Extensive experiments show that such a simple scheme
surpasses previous tuning baselines for both discriminate and generative tasks.
Our approach can potentially be complementary to many existing fine-tuning
methods.

通过细调滤波核元素，我们引入了一个简单的方法来改进现有的微调方法，以提取面向任务的表示，这种方法在区分和生成任务方面优于之前的微调基准。