Diffusion models based on Multi-Head Attention (MHA) have become ubiquitous to generate high quality images and videos. However, encoding an image or a video as a sequence of patches results in costly attention patterns, as the requirements both in terms of memory and compute grow quadratically. To alleviate this problem, we propose a drop-in replacement for MHA called the Polynomial Mixer (PoM) that has the benefit of encoding the entire sequence into an explicit state. PoM has a linear complexity with respect to the number of tokens. This explicit state also allows us to generate frames in a sequential fashion, minimizing memory and compute requirement, while still being able to train in parallel. We show the Polynomial Mixer is a universal sequence-to-sequence approximator, just like regular MHA. We adapt several Diffusion Transformers (DiT) for generating images and videos with PoM replacing MHA, and we obtain high quality samples while using less computational resources. The code is available at https://github.com/davidpicard/HoMM.

本研究解决了基于多头注意力（MHA）的扩散模型在图像和视频生成中的高内存和计算需求问题。我们提出了一个名为多项式混合器（PoM）的替代方案，其具有线性复杂性，并能以显式状态编码整个序列，显著降低资源消耗。我们的实验表明，PoM在生成高质量样本方面有效，同时降低了计算资源的需求。

PoM：使用多项式混合器的高效图像和视频生成