The painting process of artists is inherently stepwise and varies
significantly among different painters and styles. Generating detailed,
step-by-step painting processes is essential for art education and research,
yet remains largely underexplored. Traditional stroke-based rendering methods
break down images into sequences of brushstrokes, yet they fall short of
replicating the authentic processes of artists, with limitations confined to
basic brushstroke modifications. Text-to-image models utilizing diffusion
processes generate images through iterative denoising, also diverge
substantially from artists' painting process. To address these challenges, we
introduce ProcessPainter, a text-to-video model that is initially pre-trained
on synthetic data and subsequently fine-tuned with a select set of artists'
painting sequences using the LoRA model. This approach successfully generates
painting processes from text prompts for the first time. Furthermore, we
introduce an Artwork Replication Network capable of accepting arbitrary-frame
input, which facilitates the controlled generation of painting processes,
decomposing images into painting sequences, and completing semi-finished
artworks. This paper offers new perspectives and tools for advancing art
education and image generation technology.

通过合成数据预训练，在一组艺术家的绘画序列的细微调整下进行微调的文本到视频模型 ProcessPainter 首次成功地从文本提示生成了绘画过程。此外，引入了 Artwork Replication Network，能接受任意帧输入，促进了绘画过程的可控生成和艺术品的复制。本文提供了推动艺术教育和图像生成技术发展的新视角和工具。

ProcessPainter: 从序列数据中学习绘画过程

ProcessPainter: Learn Painting Process from Sequence Data

The process of painting fosters creativity and rational planning. However,
existing generative AI mostly focuses on producing visually pleasant artworks,
without emphasizing the painting process. We introduce a novel task,
Collaborative Neural Painting (CNP), to facilitate collaborative art painting
generation between humans and machines. Given any number of user-input
brushstrokes as the context or just the desired object class, CNP should
produce a sequence of strokes supporting the completion of a coherent painting.
Importantly, the process can be gradual and iterative, so allowing users'
modifications at any phase until the completion. Moreover, we propose to solve
this task using a painting representation based on a sequence of parametrized
strokes, which makes it easy both editing and composition operations. These
parametrized strokes are processed by a Transformer-based architecture with a
novel attention mechanism to model the relationship between the input strokes
and the strokes to complete. We also propose a new masking scheme to reflect
the interactive nature of CNP and adopt diffusion models as the basic learning
process for its effectiveness and diversity in the generative field. Finally,
to develop and validate methods on the novel task, we introduce a new dataset
of painted objects and an evaluation protocol to benchmark CNP both
quantitatively and qualitatively. We demonstrate the effectiveness of our
approach and the potential of the CNP task as a promising avenue for future
research.

协作神经绘画是一个新的任务，目标是支持人机共同完成连贯绘画的过程，以一系列参数化的笔触作为绘画表示，在图像生成领域中采用扩散模型作为基本学习过程，并提供了一个新的数据集和评估协议来验证该方法的有效性和潜力。