Sampling viable 3D structures (e.g., molecules and point clouds) with
SE(3)-invariance using diffusion-based models proved promising in a variety of
real-world applications, wherein SE(3)-invariant properties can be naturally
characterized by the inter-point distance manifold. However, due to the
non-trivial geometry, we still lack a comprehensive understanding of the
diffusion mechanism within such SE(3)-invariant space. This study addresses
this gap by mathematically delineating the diffusion mechanism under
SE(3)-invariance, via zooming into the interaction behavior between coordinates
and the inter-point distance manifold through the lens of differential
geometry. Upon this analysis, we propose accurate and projection-free diffusion
SDE and ODE accordingly. Such formulations enable enhancing the performance and
the speed of generation pathways; meanwhile offering valuable insights into
other systems incorporating SE(3)-invariance.

通过微分几何的视角，本研究对 SE (3) 不变空间中的扩散机制进行了数学界定，揭示了坐标与点间距流形之间的相互作用行为，并提出了准确且无需投影的扩散 SDE 和 ODE 公式，从而提高了生成路径的性能和速度，并为其他包含 SE (3) 不变性的系统提供了有价值的见解。

SE (3) 不变空间中的扩散过程

On Diffusion Process in SE(3)-invariant Space

The objective of this work is to extract target speaker's voice from a
mixture of voices using visual cues. Existing works on audio-visual speech
separation have demonstrated their performance with promising intelligibility,
but maintaining naturalness remains a challenge. To address this issue, we
propose AVDiffuSS, an audio-visual speech separation model based on a diffusion
mechanism known for its capability in generating natural samples. For an
effective fusion of the two modalities for diffusion, we also propose a
cross-attention-based feature fusion mechanism. This mechanism is specifically
tailored for the speech domain to integrate the phonetic information from
audio-visual correspondence in speech generation. In this way, the fusion
process maintains the high temporal resolution of the features, without
excessive computational requirements. We demonstrate that the proposed
framework achieves state-of-the-art results on two benchmarks, including
VoxCeleb2 and LRS3, producing speech with notably better naturalness.

本文介绍了 AVDiffuSS，一种基于扩散机制的音视频语音分离模型，通过视觉线索从声音混合中提取目标发言者的声音，该模型在保持自然性方面具有挑战，并提出了一种基于交叉注意力的特征融合机制，以实现两种模态的有效融合，并在语音生成中集成语音视觉对应的语音信息，通过该提出的框架在 VoxCeleb2 和 LRS3 这两个基准测试上取得了最先进的结果，生成的语音具有显著更好的自然音质。

透视对话：基于扩散模型的音频 - 视觉语音分离

Seeing Through the Conversation: Audio-Visual Speech Separation based on  Diffusion Model

This work introduces "You Only Diffuse Areas" (YODA), a novel method for
partial diffusion in Single-Image Super-Resolution (SISR). The core idea is to
utilize diffusion selectively on spatial regions based on attention maps
derived from the low-resolution image and the current time step in the
diffusion process. This time-dependent targeting enables a more effective
conversion to high-resolution outputs by focusing on areas that benefit the
most from the iterative refinement process, i.e., detail-rich objects. We
empirically validate YODA by extending leading diffusion-based SISR methods SR3
and SRDiff. Our experiments demonstrate new state-of-the-art performance gains
in face and general SR across PSNR, SSIM, and LPIPS metrics. A notable finding
is YODA's stabilization effect on training by reducing color shifts, especially
when induced by small batch sizes, potentially contributing to
resource-constrained scenarios. The proposed spatial and temporal adaptive
diffusion mechanism opens promising research directions, including developing
enhanced attention map extraction techniques and optimizing inference latency
based on sparser diffusion.

提出了一种名为 “You Only Diffuse Areas”（YODA）的新方法，用于单图像超分辨率（SISR）中的局部扩散，通过根据低分辨率图像和扩散过程中的当前时间步骤得到的注意力图，在空间区域上有选择地利用扩散。这种时间依赖的定位能够更有效地将图像转换为高分辨率输出，集中于从迭代精细化过程中最受益的区域，即富含细节的对象。通过扩展领先的基于扩散的 SISR 方法 SR3 和 SRDiff 来实证验证了 YODA，并且实验结果在面部和一般 SR 上通过 PSNR、SSIM 和 LPIPS 指标取得了新的最优性能提升。值得注意的发现是 YODA 对训练的稳定化效应，通过减少颜色偏移，特别是在小批量大小引起的情况下，可能有助于资源受限的场景。所提出的空间和时间自适应扩散机制开启了有关发展增强的注意力图提取技术和基于稀疏扩散的推理延迟优化的有望研究方向。