Language-based colorization produces plausible and visually pleasing colors
under the guidance of user-friendly natural language descriptions. Previous
methods implicitly assume that users provide comprehensive color descriptions
for most of the objects in the image, which leads to suboptimal performance. In
this paper, we propose a unified model to perform language-based colorization
with any-level descriptions. We leverage the pretrained cross-modality
generative model for its robust language understanding and rich color priors to
handle the inherent ambiguity of any-level descriptions. We further design
modules to align with input conditions to preserve local spatial structures and
prevent the ghosting effect. With the proposed novel sampling strategy, our
model achieves instance-aware colorization in diverse and complex scenarios.
Extensive experimental results demonstrate our advantages of effectively
handling any-level descriptions and outperforming both language-based and
automatic colorization methods. The code and pretrained models are available
at: this https URL

本文提出了一种利用自然语言描述进行基于语言的彩色生成，通过与预训练的跨模态生成模型相结合，实现了对任意级别的颜色描述的处理，能够有效处理复杂环境下的样例感知着色。