In the last few years, Neural Painting (NP) techniques became capable of
producing extremely realistic artworks. This paper advances the state of the
art in this emerging research domain by proposing the first approach for
Interactive NP. Considering a setting where a user looks at a scene and tries
to reproduce it on a painting, our objective is to develop a computational
framework to assist the users creativity by suggesting the next strokes to
paint, that can be possibly used to complete the artwork. To accomplish such a
task, we propose I-Paint, a novel method based on a conditional transformer
Variational AutoEncoder (VAE) architecture with a two-stage decoder. To
evaluate the proposed approach and stimulate research in this area, we also
introduce two novel datasets. Our experiments show that our approach provides
good stroke suggestions and compares favorably to the state of the art.
Additional details, code and examples are available at
this https URL

通过提出一种基于条件变换自动编码器 (VAE) 架构的双阶段解码器的创新方法 I-Paint，本研究旨在为用户在绘画过程中提供下一笔建议，以帮助提高创造力，并且通过引入两个新的数据集进行评估和刺激相关领域的研究。实验结果表明该方法提供良好的笔划建议，并与现有技术相比表现出优势。

交互式神经绘画

Interactive Neural Painting

Neural painting refers to the procedure of producing a series of strokes for
a given image and non-photo-realistically recreating it using neural networks.
While reinforcement learning (RL) based agents can generate a stroke sequence
step by step for this task, it is not easy to train a stable RL agent. On the
other hand, stroke optimization methods search for a set of stroke parameters
iteratively in a large search space; such low efficiency significantly limits
their prevalence and practicality. Different from previous methods, in this
paper, we formulate the task as a set prediction problem and propose a novel
Transformer-based framework, dubbed Paint Transformer, to predict the
parameters of a stroke set with a feed forward network. This way, our model can
generate a set of strokes in parallel and obtain the final painting of size 512
* 512 in near real time. More importantly, since there is no dataset available
for training the Paint Transformer, we devise a self-training pipeline such
that it can be trained without any off-the-shelf dataset while still achieving
excellent generalization capability. Experiments demonstrate that our method
achieves better painting performance than previous ones with cheaper training
and inference costs. Codes and models are available.

本文提出了一种名为 Paint Transformer 的基于 Transformer 结构的神经网络框架，通过一个前馈网络来预测笔画集合的参数，从而实现在短时间内生成一个 512 * 512 的绘画。此外，作者设计了一个自学习流程，使其可以在没有任何预训练数据的情况下进行训练，并取得了比以往更好的绘画性能。