Personalization techniques for large text-to-image (T2I) models allow users
to incorporate new concepts from reference images. However, existing methods
primarily rely on textual descriptions, leading to limited control over
customized images and failing to support fine-grained and local editing (e.g.,
shape, pose, and details). In this paper, we identify sketches as an intuitive
and versatile representation that can facilitate such control, e.g., contour
lines capturing shape information and flow lines representing texture. This
motivates us to explore a novel task of sketch concept extraction: given one or
more sketch-image pairs, we aim to extract a special sketch concept that
bridges the correspondence between the images and sketches, thus enabling
sketch-based image synthesis and editing at a fine-grained level. To accomplish
this, we introduce CustomSketching, a two-stage framework for extracting novel
sketch concepts. Considering that an object can often be depicted by a contour
for general shapes and additional strokes for internal details, we introduce a
dual-sketch representation to reduce the inherent ambiguity in sketch
depiction. We employ a shape loss and a regularization loss to balance fidelity
and editability during optimization. Through extensive experiments, a user
study, and several applications, we show our method is effective and superior
to the adapted baselines.

使用个性化技术的大型文本到图像（T2I）模型允许用户从参考图像中融入新概念。本文旨在通过探索一种新颖的任务，即草图概念提取，通过两阶段框架 CustomSketching 实现草图概念的提取，以实现基于草图的图像合成和编辑。

CustomSketching：基于草图概念提取的草图图像合成和编辑

CustomSketching: Sketch Concept Extraction for Sketch-based Image  Synthesis and Editing

Large text-to-image models have revolutionized the ability to generate
imagery using natural language. However, particularly unique or personal visual
concepts, such as your pet, an object in your house, etc., will not be captured
by the original model. This has led to interest in how to inject new visual
concepts, bound to a new text token, using as few as 4-6 examples. Despite
significant progress, this task remains a formidable challenge, particularly in
preserving the subject's identity. While most researchers attempt to to address
this issue by modifying model architectures, our approach takes a data-centric
perspective, advocating the modification of data rather than the model itself.
We introduce a novel regularization dataset generation strategy on both the
text and image level; demonstrating the importance of a rich and structured
regularization dataset (automatically generated) to prevent losing text
coherence and better identity preservation. The better quality is enabled by
allowing up to 5x more fine-tuning iterations without overfitting and
degeneration. The generated renditions of the desired subject preserve even
fine details such as text and logos; all while maintaining the ability to
generate diverse samples that follow the input text prompt. Since our method
focuses on data augmentation, rather than adjusting the model architecture, it
is complementary and can be combined with prior work. We show on established
benchmarks that our data-centric approach forms the new state of the art in
terms of image quality, with the best trade-off between identity preservation,
diversity, and text alignment.

使用大型文本到图像模型生成图像的能力已经引起了巨大的变革，但是对于特定独特或个人化的视觉概念，如您的宠物、屋内物品等，并不能被原始模型捕捉到。本文通过采用数据为中心的方法，提出了一种新颖的正则化数据集生成策略，旨在解决文本连贯性丧失和身份保留问题，进一步提高图像质量，并能生成符合输入文本提示的多样样本。实验证明，我们的数据为中心的方法在图像质量方面具有新的技术实力，在身份保留、多样性和文本对齐之间取得了最佳的平衡。