Current image captioning works usually focus on generating descriptions in an
autoregressive manner. However, there are limited works that focus on
generating descriptions non-autoregressively, which brings more decoding
diversity. Inspired by the success of diffusion models on generating
natural-looking images, we propose a novel method DiffCap to apply continuous
diffusions on image captioning. Unlike image generation where the output is
fixed-size and continuous, image description length varies with discrete
tokens. Our method transforms discrete tokens in a natural way and applies
continuous diffusion on them to successfully fuse extracted image features for
diffusion caption generation. Our experiments on COCO dataset demonstrate that
our method uses a much simpler structure to achieve comparable results to the
previous non-autoregressive works. Apart from quality, an intriguing property
of DiffCap is its high diversity during generation, which is missing from many
autoregressive models. We believe our method on fusing multimodal features in
diffusion language generation will inspire more researches on multimodal
language generation tasks for its simplicity and decoding flexibility.

本文提出了一种基于连续扩散的新方法 ——DiffCap，用于图像标题生成中的多模态特征融合。相比于自回归式的方法，DiffCap 采用了非自回归式方法，可在生成时提供更高的多样性与灵活性。实验结果表明，我们的方法在保证生成品质的同时，使用了更加简单的结构，这一设计将会启发更多关于多模态生成任务方面的研究。