We introduce a novel sketch-to-image tool that aligns with the iterative
refinement process of artists. Our tool lets users sketch blocking strokes to
coarsely represent the placement and form of objects and detail strokes to
refine their shape and silhouettes. We develop a two-pass algorithm for
generating high-fidelity images from such sketches at any point in the
iterative process. In the first pass we use a ControlNet to generate an image
that strictly follows all the strokes (blocking and detail) and in the second
pass we add variation by renoising regions surrounding blocking strokes. We
also present a dataset generation scheme that, when used to train a ControlNet
architecture, allows regions that do not contain strokes to be interpreted as
not-yet-specified regions rather than empty space. We show that this
partial-sketch-aware ControlNet can generate coherent elements from partial
sketches that only contain a small number of strokes. The high-fidelity images
produced by our approach serve as scaffolds that can help the user adjust the
shape and proportions of objects or add additional elements to the composition.
We demonstrate the effectiveness of our approach with a variety of examples and
evaluative comparisons.

我们介绍了一种与艺术家的迭代细化过程相适应的新型素描到图像工具。我们的工具允许用户绘制出粗略表示物体放置和形状的阻塞笔触以及用于精炼形状和轮廓的详细笔触。我们开发了一个两步算法，从这些素描中的任意时刻生成高保真度图像。第一步中，我们使用 ControlNet 生成完全遵循所有笔触（阻塞笔触和详细笔触）的图像，第二步通过给阻塞笔触周围的区域添加噪声来增加变异性。我们还提出了一种数据集生成方案，用于训练 ControlNet 架构，使不包含笔触的区域被解释为尚未指定的区域而不是空白空间。我们展示了这种部分素描感知的 ControlNet 能够从仅含少量笔触的部分素描中生成连贯的元素。我们的方法生成的高保真度图像可以作为脚手架，帮助用户调整物体的形状和比例，或者在构图中添加额外的元素。我们通过各种例子和评估性比较证明了我们方法的有效性。

区块与细节：搭建草图到图像生成系统

Block and Detail: Scaffolding Sketch-to-Image Generation

We propose Easy End-to-End Diffusion-based Text to Speech, a simple and
efficient end-to-end text-to-speech model based on diffusion. E3 TTS directly
takes plain text as input and generates an audio waveform through an iterative
refinement process. Unlike many prior work, E3 TTS does not rely on any
intermediate representations like spectrogram features or alignment
information. Instead, E3 TTS models the temporal structure of the waveform
through the diffusion process. Without relying on additional conditioning
information, E3 TTS could support flexible latent structure within the given
audio. This enables E3 TTS to be easily adapted for zero-shot tasks such as
editing without any additional training. Experiments show that E3 TTS can
generate high-fidelity audio, approaching the performance of a state-of-the-art
neural TTS system. Audio samples are available at this https URL

我们提出了一种简单高效的端到端扩散式文本转语音模型，通过扩散过程直接从纯文本生成音频波形。该模型无需中间表示，能够支持给定音频的灵活潜在结构，从而实现轻松适应零样本任务。实验证明，该模型能够生成高保真音频，接近最先进的神经网络语音合成系统的性能。