Recently, diffusion models have demonstrated their effectiveness in
generating extremely high-quality images and have found wide-ranging
applications, including automatic sketch colorization. However, most existing
models use text to guide the conditional generation, with fewer attempts
exploring the potential advantages of using image tokens as conditional inputs
for networks. As such, this paper exhaustively investigates image-guided
models, specifically targeting reference-based sketch colorization, which aims
to colorize sketch images using reference color images. We investigate three
critical aspects of reference-based diffusion models: the shortcomings compared
to text-based counterparts, the training strategies, and the capability in
zero-shot, sequential text-based manipulation. We introduce two variations of
an image-guided latent diffusion model using different image tokens from the
pre-trained CLIP image encoder, and we propose corresponding manipulation
methods to adjust their results sequentially using weighted text inputs. We
conduct comprehensive evaluations of our models through qualitative and
quantitative experiments, as well as a user study.

该研究论文探讨了基于图像引导的扩散模型在参考图像引导下的素描着色，并提出了两种使用不同图像特征作为条件输入的变种隐式扩散模型，以及相应的调整结果方法。通过定性和定量实验以及用户研究，全面评估了这些模型的性能。