In this paper, we propose ``SimVLG'', a streamlined framework for the
pre-training of computationally intensive vision-language generative models,
leveraging frozen pre-trained large language models (LLMs). The prevailing
paradigm in vision-language pre-training (VLP) typically involves a two-stage
optimization process: an initial resource-intensive phase dedicated to
general-purpose vision-language representation learning, aimed at extracting
and consolidating pertinent visual features, followed by a subsequent phase
focusing on end-to-end alignment between visual and linguistic modalities. Our
one-stage, single-loss framework circumvents the aforementioned computationally
demanding first stage of training by gradually merging similar visual tokens
during training. This gradual merging process effectively compacts the visual
information while preserving the richness of semantic content, leading to fast
convergence without sacrificing performance. Our experiments show that our
approach can speed up the training of vision-language models by a factor
$\times 5$ without noticeable impact on the overall performance. Additionally,
we show that our models can achieve comparable performance to current
vision-language models with only $1/10$ of the data. Finally, we demonstrate
how our image-text models can be easily adapted to video-language generative
tasks through a novel soft attentive temporal token merging modules.

本文提出了 “SimVLG”—— 一种用于预训练计算密集型视觉语言生成模型的简化框架，利用冻结的预训练大型语言模型 (LLMs)。我们的单阶段单损失框架通过在训练过程中逐渐合并相似的视觉标记来绕过计算密集型的初始训练阶段，从而实现了快速收敛而不损失性能。实验证明，我们的方法可以将视觉语言模型的训练速度提高 5 倍，而整体性能几乎没有明显影响。此外，我们展示了我们的模型只需原数据的十分之一即可达到与当前视觉语言模型相当的性能。最后，我们通过一种新颖的软性注意力时序标记合并模块展示了如何轻松将我们的图像 - 文本模型应用于视频 - 语言生成任务。