Video diffusion models have recently made great progress in generation
quality, but are still limited by the high memory and computational
requirements. This is because current video diffusion models often attempt to
process high-dimensional videos directly. To tackle this issue, we propose
content-motion latent diffusion model (CMD), a novel efficient extension of
pretrained image diffusion models for video generation. Specifically, we
propose an autoencoder that succinctly encodes a video as a combination of a
content frame (like an image) and a low-dimensional motion latent
representation. The former represents the common content, and the latter
represents the underlying motion in the video, respectively. We generate the
content frame by fine-tuning a pretrained image diffusion model, and we
generate the motion latent representation by training a new lightweight
diffusion model. A key innovation here is the design of a compact latent space
that can directly utilizes a pretrained image diffusion model, which has not
been done in previous latent video diffusion models. This leads to considerably
better quality generation and reduced computational costs. For instance, CMD
can sample a video 7.7$\times$ faster than prior approaches by generating a
video of 512$\times$1024 resolution and length 16 in 3.1 seconds. Moreover, CMD
achieves an FVD score of 212.7 on WebVid-10M, 27.3% better than the previous
state-of-the-art of 292.4.

该研究提出了一种内容 - 动态潜在扩散模型 (CMD)，它是针对视频生成的预训练图像扩散模型的一种高效扩展。CMD 通过使用预训练图像扩散模型生成内容帧和训练轻量级扩散模型生成动态潜在表征，实现了更高质量的视频生成和降低的计算成本。

通过内容 - 帧运动 - 潜变分解实现高效视频扩散模型

Efficient Video Diffusion Models via Content-Frame Motion-Latent  Decomposition

The remarkable capabilities of pretrained image diffusion models have been
utilized not only for generating fixed-size images but also for creating
panoramas. However, naive stitching of multiple images often results in visible
seams. Recent techniques have attempted to address this issue by performing
joint diffusions in multiple windows and averaging latent features in
overlapping regions. However, these approaches, which focus on seamless montage
generation, often yield incoherent outputs by blending different scenes within
a single image. To overcome this limitation, we propose SyncDiffusion, a
plug-and-play module that synchronizes multiple diffusions through gradient
descent from a perceptual similarity loss. Specifically, we compute the
gradient of the perceptual loss using the predicted denoised images at each
denoising step, providing meaningful guidance for achieving coherent montages.
Our experimental results demonstrate that our method produces significantly
more coherent outputs compared to previous methods (66.35% vs. 33.65% in our
user study) while still maintaining fidelity (as assessed by GIQA) and
compatibility with the input prompt (as measured by CLIP score).

本文提出了 SyncDiffusion 方法，使用感知相似性损失从梯度下降同步多重扩散，用于生成更加协调的全景图。

通过同步联合扩散实现一致的蒙太奇

SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions

Recent breakthroughs in text-to-image synthesis have been driven by diffusion
models trained on billions of image-text pairs. Adapting this approach to 3D
synthesis would require large-scale datasets of labeled 3D data and efficient
architectures for denoising 3D data, neither of which currently exist. In this
work, we circumvent these limitations by using a pretrained 2D text-to-image
diffusion model to perform text-to-3D synthesis. We introduce a loss based on
probability density distillation that enables the use of a 2D diffusion model
as a prior for optimization of a parametric image generator. Using this loss in
a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a
Neural Radiance Field, or NeRF) via gradient descent such that its 2D
renderings from random angles achieve a low loss. The resulting 3D model of the
given text can be viewed from any angle, relit by arbitrary illumination, or
composited into any 3D environment. Our approach requires no 3D training data
and no modifications to the image diffusion model, demonstrating the
effectiveness of pretrained image diffusion models as priors.

本文提出了一种使用基于文本转图像的扩散模型进行文本到 3D 合成的方法，该方法绕过了需要大规模标记的 3D 数据集和能够去噪的 3D 数据的限制，将 2D 的扩散模型作为先验，通过梯度下降优化 3D 模型（Neural Radiance Field），并使用概率密度蒸馏引入的损失函数将 2D 扩散模型与 3D 模型相结合。这种方法不需要 3D 训练数据，也不需要修改图像扩散模型，证明了使用预训练的图像扩散模型作为先验的有效性。