Recent years have seen significant advancements in foundation models through generative pre-training, yet algorithmic innovation in this space has largely stagnated around autoregressive models for discrete signals and diffusion models for continuous signals. This stagnation creates a bottleneck that prevents us from fully unlocking the potential of rich multi-modal data, which in turn limits the progress on multimodal intelligence. We argue that an inference-first perspective, which prioritizes scaling efficiency during inference time across sequence length and refinement steps, can inspire novel generative pre-training algorithms. Using Inductive Moment Matching (IMM) as a concrete example, we demonstrate how addressing limitations in diffusion models' inference process through targeted modifications yields a stable, single-stage algorithm that achieves superior sample quality with over an order of magnitude greater inference efficiency.

本研究解决了生成预训练领域中关于自回归模型和扩散模型的算法创新停滞的问题。提出采用以推理为中心的方法，优先考虑推理时间的规模效率，从而激发新的生成预训练算法。通过实例展示，该方法能在提高样本质量的同时，显著提升推理效率，推动多模态智能的发展。

推理时间规模化的思路如何促进生成预训练算法