Training diffusion models for audiovisual sequences allows for a range of
generation tasks by learning conditional distributions of various input-output
combinations of the two modalities. Nevertheless, this strategy often requires
training a separate model for each task which is expensive. Here, we propose a
novel training approach to effectively learn arbitrary conditional
distributions in the audiovisual space.Our key contribution lies in how we
parameterize the diffusion timestep in the forward diffusion process. Instead
of the standard fixed diffusion timestep, we propose applying variable
diffusion timesteps across the temporal dimension and across modalities of the
inputs. This formulation offers flexibility to introduce variable noise levels
for various portions of the input, hence the term mixture of noise levels. We
propose a transformer-based audiovisual latent diffusion model and show that it
can be trained in a task-agnostic fashion using our approach to enable a
variety of audiovisual generation tasks at inference time. Experiments
demonstrate the versatility of our method in tackling cross-modal and
multimodal interpolation tasks in the audiovisual space. Notably, our proposed
approach surpasses baselines in generating temporally and perceptually
consistent samples conditioned on the input. Project page: avdit2024.github.io

通过使用新的训练方法和变量扩散时间步长，我们提出了一种基于转换器的音视频潜在扩散模型，可在任务不可知的情况下进行训练，并在推理过程中实现各种音视频生成任务，克服基线模型在生成条件输入上的时间和感知上的不连贯样本的局限性。