Most existing multimodality methods use separate backbones for autoregression-based discrete text generation and diffusion-based continuous visual generation, or the same backbone by discretizing the visual data to use autoregression for both text and visual generation. In this paper, we propose to study a simple idea: share one transformer for both autoregression and diffusion. The feasibility comes from two main aspects: (i) Transformer is successfully applied to diffusion for visual generation, and (ii) transformer training for autoregression and diffusion is very similar, and the difference merely lies in that diffusion uses bidirectional attention mask and autoregression uses causal attention mask. Experimental results show that our approach achieves comparable image generation performance to current state-of-the-art methods as well as maintains the text generation capability. The project is publicly available at https://monoformer.github.io/.

本研究针对现有多模态方法中自回归文本生成与扩散视觉生成使用分离主干的问题，提出了共享单一变压器的创新思路。研究表明，该方法在图像生成性能上可与当前最先进的方法媲美，同时保留文本生成能力，具有广泛的应用潜力。

MonoFormer：一个变压器同时用于扩散和自回归