While Low-Rank Adaptation (LoRA) has proven beneficial for efficiently
fine-tuning large models, LoRA fine-tuned text-to-image diffusion models lack
diversity in the generated images, as the model tends to copy data from the
observed training samples. This effect becomes more pronounced at higher values
of adapter strength and for adapters with higher ranks which are fine-tuned on
smaller datasets. To address these challenges, we present FouRA, a novel
low-rank method that learns projections in the Fourier domain along with
learning a flexible input-dependent adapter rank selection strategy. Through
extensive experiments and analysis, we show that FouRA successfully solves the
problems related to data copying and distribution collapse while significantly
improving the generated image quality. We demonstrate that FouRA enhances the
generalization of fine-tuned models thanks to its adaptive rank selection. We
further show that the learned projections in the frequency domain are
decorrelated and prove effective when merging multiple adapters. While FouRA is
motivated for vision tasks, we also demonstrate its merits for language tasks
on the GLUE benchmark.

FouRA 是一种新颖的低秩方法，通过学习傅里叶域的投影以及灵活的基于输入的适配器秩选择策略，成功解决了数据复制和分布坍塌问题，并显著提高了生成图像的质量。