Language-based colorization produces plausible and visually pleasing colors
under the guidance of user-friendly natural language descriptions. Previous
methods implicitly assume that users provide comprehensive color descriptions
for most of the objects in the image, which leads to suboptimal performance. In
this paper, we propose a unified model to perform language-based colorization
with any-level descriptions. We leverage the pretrained cross-modality
generative model for its robust language understanding and rich color priors to
handle the inherent ambiguity of any-level descriptions. We further design
modules to align with input conditions to preserve local spatial structures and
prevent the ghosting effect. With the proposed novel sampling strategy, our
model achieves instance-aware colorization in diverse and complex scenarios.
Extensive experimental results demonstrate our advantages of effectively
handling any-level descriptions and outperforming both language-based and
automatic colorization methods. The code and pretrained models are available
at: this https URL

本文提出了一种利用自然语言描述进行基于语言的彩色生成，通过与预训练的跨模态生成模型相结合，实现了对任意级别的颜色描述的处理，能够有效处理复杂环境下的样例感知着色。

L-CAD: 基于语言的任意级别描述着色

L-CAD: Language-based Colorization with Any-level Descriptions

Image colorization is inherently an ill-posed problem with multi-modal
uncertainty. Previous methods leverage the deep neural network to map input
grayscale images to plausible color outputs directly. Although these
learning-based methods have shown impressive performance, they usually fail on
the input images that contain multiple objects. The leading cause is that
existing models perform learning and colorization on the entire image. In the
absence of a clear figure-ground separation, these models cannot effectively
locate and learn meaningful object-level semantics. In this paper, we propose a
method for achieving instance-aware colorization. Our network architecture
leverages an off-the-shelf object detector to obtain cropped object images and
uses an instance colorization network to extract object-level features. We use
a similar network to extract the full-image features and apply a fusion module
to full object-level and image-level features to predict the final colors. Both
colorization networks and fusion modules are learned from a large-scale
dataset. Experimental results show that our work outperforms existing methods
on different quality metrics and achieves state-of-the-art performance on image
colorization.

本论文提出了一种基于物体实例的彩色化方法，利用深度神经网络提取物体级别的特征，并采用融合模块组合全图像级别和物体级别的特征来预测最终颜色，大规模实验结果表明该方法优于现有的方法，达到了图像彩色化的最新性能。