This project report summarizes our journey to perform stable diffusion
fine-tuning on a dataset containing Calvin and Hobbes comics. The purpose is to
convert any given input image into the comic style of Calvin and Hobbes,
essentially performing style transfer. We train stable-diffusion-v1.5 using Low
Rank Adaptation (LoRA) to efficiently speed up the fine-tuning process. The
diffusion itself is handled by a Variational Autoencoder (VAE), which is a
U-net. Our results were visually appealing for the amount of training time and
the quality of input data that went into training.

本研究提出了一种基于扩散微调的方法，在包含卡尔文和霍布斯漫画的数据集上进行样式转换，通过使用低秩调整（LoRA）训练稳定扩散版本 1.5 来加速微调过程，扩散本身由变分自动编码器（VAE）来处理，实验结果表明在相对较少的训练时间和优质输入数据的情况下能够获得视觉上吸引人的结果。

使用稳定扩散进行卡尔文与霍布斯连环漫画的风格转换

Style Transfer to Calvin and Hobbes comics using Stable Diffusion

Recent work using pretrained transformers has shown impressive performance
when fine-tuned with data from the downstream problem of interest. However,
they struggle to retain that performance when the data characteristics changes.
In this paper, we focus on continual learning, where a pre-trained transformer
is updated to perform well on new data, while retaining its performance on data
it was previously trained on. Earlier works have tackled this primarily through
methods inspired from prompt tuning. We question this choice, and investigate
the applicability of Low Rank Adaptation (LoRA) to continual learning. On a
range of domain-incremental learning benchmarks, our LoRA-based solution,
CoLoR, yields state-of-the-art performance, while still being as parameter
efficient as the prompt tuning based methods.

本文主要研究了基于预训练转换器的领域增量学习问题，在课题涉及的下游数据上，该方法表现出了令人印象深刻的性能，但当数据特征发生变化时，性能会下降；通过研究 Low Rank Adaptation（LoRA）在领域增量学习中的适用性，我们的基于 LoRA 的解决方案 CoLoR 在一系列领域增量学习基准测试中取得了最先进的性能，同时仍然与基于提示调整的方法一样参数高效。