We propose a framework for automatic colorization that allows for iterative
editing and modifications. The core of our framework lies in an imagination
module: by understanding the content within a grayscale image, we utilize a
pre-trained image generation model to generate multiple images that contain the
same content. These images serve as references for coloring, mimicking the
process of human experts. As the synthesized images can be imperfect or
different from the original grayscale image, we propose a Reference Refinement
Module to select the optimal reference composition. Unlike most previous
end-to-end automatic colorization algorithms, our framework allows for
iterative and localized modifications of the colorization results because we
explicitly model the coloring samples. Extensive experiments demonstrate the
superiority of our framework over existing automatic colorization algorithms in
editability and flexibility. Project page:
this https URL

我们提出了一个自动上色的框架，具有迭代编辑和修改的能力，并利用一个想象模块通过理解灰度图像中的内容，利用预训练图像生成模型生成包含相同内容的多个图像用于上色参考，模仿人类专家的过程；我们使用参考修正模块来选择最佳的参考合成；与大多数现有的自动上色算法不同，我们的框架允许对上色结果进行迭代和局部修改，因为我们明确地对上色样本进行建模；大量实验证明了我们的框架在可编辑性和灵活性方面优于现有的自动上色算法。

基于想象力的自动可控彩色化

Automatic Controllable Colorization via Imagination

Foundation models encompass an extensive knowledge base and offer remarkable
transferability. However, this knowledge becomes outdated or insufficient over
time. The challenge lies in updating foundation models to accommodate novel
information while retaining their original ability. In this paper, we present a
novel approach to achieving continual model updates by effecting localized
modifications to a small subset of parameters. Guided by insights gleaned from
prior analyses of foundational models, we first localize a specific layer for
model refinement and then introduce an importance scoring mechanism designed to
update only the most crucial weights. Our method is exhaustively evaluated on
foundational vision-language models, measuring its efficacy in both learning
new information and preserving pre-established knowledge across a diverse
spectrum of continual learning tasks, including Aircraft, Birdsnap CIFAR-100,
CUB, Cars, and GTSRB. The results show that our method improves the existing
continual learning methods by 0.5\% - 10\% on average, and reduces the loss of
pre-trained knowledge from around 5\% to 0.97\%. Comprehensive ablation studies
substantiate our method design, shedding light on the contributions of each
component to controllably learning new knowledge and mitigating the forgetting
of pre-trained knowledge.

本文介绍了一种更新基础模型以适应新信息并保留其原始能力的新方法，通过对一小部分参数进行本地化修改，并引入重要性评分机制来仅更新最关键的权重，在多样的持续学习任务中得到了全面评估，表明其能够提高现有的持续学习方法并减少已训练知识的损失。