We present ActionDiffusion -- a novel diffusion model for procedure planning
in instructional videos that is the first to take temporal inter-dependencies
between actions into account in a diffusion model for procedure planning. This
approach is in stark contrast to existing methods that fail to exploit the rich
information content available in the particular order in which actions are
performed. Our method unifies the learning of temporal dependencies between
actions and denoising of the action plan in the diffusion process by projecting
the action information into the noise space. This is achieved 1) by adding
action embeddings in the noise masks in the noise-adding phase and 2) by
introducing an attention mechanism in the noise prediction network to learn the
correlations between different action steps. We report extensive experiments on
three instructional video benchmark datasets (CrossTask, Coin, and NIV) and
show that our method outperforms previous state-of-the-art methods on all
metrics on CrossTask and NIV and all metrics except accuracy on Coin dataset.
We show that by adding action embeddings into the noise mask the diffusion
model can better learn action temporal dependencies and increase the
performances on procedure planning.

我们提出了一个新颖的扩散模型 ActionDiffusion，用于指导教学视频中的过程规划，它是第一个在扩散模型中考虑动作之间的时间依赖关系，与现有方法相比，利用了动作按特定顺序执行的丰富信息内容。我们的方法通过将动作信息投影到噪声空间实现了对动作的学习和扩散过程中的去噪，其中，在噪声添加阶段通过在噪声掩码中添加动作嵌入，并通过在噪声预测网络中引入注意机制来学习不同动作步骤之间的相关性。我们在三个教学视频基准数据集（CrossTask、Coin 和 NIV）上进行了广泛的实验，并展示了我们的方法在 CrossTask 和 NIV 的所有指标以及 Coin 数据集上的除准确性以外的所有指标上胜过了以前的最先进方法。我们的研究结果表明，通过将动作嵌入添加到噪声掩码中，扩散模型能够更好地学习动作的时间依赖关系，并提高了过程规划的性能。