Score distillation sampling (SDS) has proven to be an important tool,
enabling the use of large-scale diffusion priors for tasks operating in
data-poor domains. Unfortunately, SDS has a number of characteristic artifacts
that limit its usefulness in general-purpose applications. In this paper, we
make progress toward understanding the behavior of SDS and its variants by
viewing them as solving an optimal-cost transport path from a source
distribution to a target distribution. Under this new interpretation, these
methods seek to transport corrupted images (source) to the natural image
distribution (target). We argue that current methods' characteristic artifacts
are caused by (1) linear approximation of the optimal path and (2) poor
estimates of the source distribution. We show that calibrating the text
conditioning of the source distribution can produce high-quality generation and
translation results with little extra overhead. Our method can be easily
applied across many domains, matching or beating the performance of specialized
methods. We demonstrate its utility in text-to-2D, text-based NeRF
optimization, translating paintings to real images, optical illusion
generation, and 3D sketch-to-real. We compare our method to existing approaches
for score distillation sampling and show that it can produce high-frequency
details with realistic colors.

通过最优成本传输路径的观点，我们提出了一种新的方法，通过校准源分布的文本条件，可以在许多领域中产生高质量的生成和翻译结果，并击败专用方法的性能。

重新考虑分数蒸馏作为图像分布之间的桥梁

Rethinking Score Distillation as a Bridge Between Image Distributions

Diffusion models, trained on large amounts of data, showed remarkable
performance for image synthesis. They have high error consistency with humans
and low texture bias when used for classification. Furthermore, prior work
demonstrated the decomposability of their bottleneck layer representations into
semantic directions. In this work, we analyze how well such representations are
aligned to human responses on a triplet odd-one-out task. We find that despite
the aforementioned observations: I) The representational alignment with humans
is comparable to that of models trained only on ImageNet-1k. II) The most
aligned layers of the denoiser U-Net are intermediate layers and not the
bottleneck. III) Text conditioning greatly improves alignment at high noise
levels, hinting at the importance of abstract textual information, especially
in the early stage of generation.

在这项工作中，我们分析了大规模数据训练的扩散模型对于三选一任务中人类响应的表示对齐情况，发现尽管其与人类响应的对齐程度与仅在 ImageNet-1k 上训练的模型相当，但去噪 U-Net 的对齐层主要为中间层而非瓶颈层，而且文本条件对于高噪声水平下的对齐显著改善，暗示了抽象文本信息的重要性，尤其是在生成的早期阶段。

潜在扩散模型的人类对齐分析

An Analysis of Human Alignment of Latent Diffusion Models

We consider the task of generating diverse and realistic videos guided by
natural audio samples from a wide variety of semantic classes. For this task,
the videos are required to be aligned both globally and temporally with the
input audio: globally, the input audio is semantically associated with the
entire output video, and temporally, each segment of the input audio is
associated with a corresponding segment of that video. We utilize an existing
text-conditioned video generation model and a pre-trained audio encoder model.
The proposed method is based on a lightweight adaptor network, which learns to
map the audio-based representation to the input representation expected by the
text-to-video generation model. As such, it also enables video generation
conditioned on text, audio, and, for the first time as far as we can ascertain,
on both text and audio. We validate our method extensively on three datasets
demonstrating significant semantic diversity of audio-video samples and further
propose a novel evaluation metric (AV-Align) to assess the alignment of
generated videos with input audio samples. AV-Align is based on the detection
and comparison of energy peaks in both modalities. In comparison to recent
state-of-the-art approaches, our method generates videos that are better
aligned with the input sound, both with respect to content and temporal axis.
We also show that videos produced by our method present higher visual quality
and are more diverse.

生成多样化和逼真的视频，根据语义类别广泛的自然音频样本进行引导。采用了基于轻量级适配器网络的方法，将音频基础表示映射到文本 - 视频生成模型所期望的输入表示，实现了对文本、音频以及文本和音频的生成视频。在三个数据集上验证了该方法，展示了生成的音频视频样本的显著语义多样性，并提出了一种新的评估度量（AV-Align）以评估生成视频与输入音频样本的对齐性。与最新的先进方法相比，我们的方法生成的视频在内容和时间轴上都与输入音频更好地对齐，并且呈现更高的视觉质量和多样性。