In this paper, we point out suboptimal noise-data mapping leads to slow
training of diffusion models. During diffusion training, current methods
diffuse each image across the entire noise space, resulting in a mixture of all
images at every point in the noise layer. We emphasize that this random mixture
of noise-data mapping complicates the optimization of the denoising function in
diffusion models. Drawing inspiration from the immiscible phenomenon in
physics, we propose Immiscible Diffusion, a simple and effective method to
improve the random mixture of noise-data mapping. In physics, miscibility can
vary according to various intermolecular forces. Thus, immiscibility means that
the mixing of the molecular sources is distinguishable. Inspired by this, we
propose an assignment-then-diffusion training strategy. Specifically, prior to
diffusing the image data into noise, we assign diffusion target noise for the
image data by minimizing the total image-noise pair distance in a mini-batch.
The assignment functions analogously to external forces to separate the
diffuse-able areas of images, thus mitigating the inherent difficulties in
diffusion training. Our approach is remarkably simple, requiring only one line
of code to restrict the diffuse-able area for each image while preserving the
Gaussian distribution of noise. This ensures that each image is projected only
to nearby noise. To address the high complexity of the assignment algorithm, we
employ a quantized-assignment method to reduce the computational overhead to a
negligible level. Experiments demonstrate that our method achieve up to 3x
faster training for consistency models and DDIM on the CIFAR dataset, and up to
1.3x faster on CelebA datasets for consistency models. Besides, we conduct
thorough analysis about the Immiscible Diffusion, which sheds lights on how it
improves diffusion training speed while improving the fidelity.

通过提出不可混合扩散方法，该研究旨在改进扩散模型的训练速度和准确性。该方法通过为图像数据分配目标噪声，限定了图像的扩散区域，同时保持噪声的高斯分布，从而实现更快的训练速度和更好的还原质量。

非混合扩散：噪声分配加速扩散训练

Immiscible Diffusion: Accelerating Diffusion Training with Noise  Assignment

This paper presents DNAct, a language-conditioned multi-task policy framework
that integrates neural rendering pre-training and diffusion training to enforce
multi-modality learning in action sequence spaces. To learn a generalizable
multi-task policy with few demonstrations, the pre-training phase of DNAct
leverages neural rendering to distill 2D semantic features from foundation
models such as Stable Diffusion to a 3D space, which provides a comprehensive
semantic understanding regarding the scene. Consequently, it allows various
applications to challenging robotic tasks requiring rich 3D semantics and
accurate geometry. Furthermore, we introduce a novel approach utilizing
diffusion training to learn a vision and language feature that encapsulates the
inherent multi-modality in the multi-task demonstrations. By reconstructing the
action sequences from different tasks via the diffusion process, the model is
capable of distinguishing different modalities and thus improving the
robustness and the generalizability of the learned representation. DNAct
significantly surpasses SOTA NeRF-based multi-task manipulation approaches with
over 30% improvement in success rate. Project website: dnact.github.io.

本文介绍了 DNAct，一种语言驱动的多任务策略框架。该框架集成了神经渲染预训练和扩散训练，以在动作序列空间中实现多模态学习。通过利用神经渲染从基础模型中提取二维语义特征并转化到三维空间，DNAct 可以对场景进行全面的语义理解，从而适用于需要丰富三维语义和准确几何信息的各种挑战性机器人任务。此外，通过扩散训练学习视觉和语言特征， DNAct 能够区分不同任务的多模态特征从而提高学习表示的鲁棒性和泛化性能。与现有的基于 NeRF 的多任务操作方法相比，DNAct 在成功率上有 30% 以上的改进。