Large Language Models (LLMs) have garnered significant attention for their
advancements in natural language processing, demonstrating unparalleled prowess
in text comprehension and generation. Yet, the simultaneous generation of
images with coherent textual narratives remains an evolving frontier. In
response, we introduce an innovative interleaved vision-and-language generation
technique anchored by the concept of "generative vokens," acting as the bridge
for harmonized image-text outputs. Our approach is characterized by a
distinctive two-staged training strategy focusing on description-free
multimodal generation, where the training requires no comprehensive
descriptions of images. To bolster model integrity, classifier-free guidance is
incorporated, enhancing the effectiveness of vokens on image generation. Our
model, MiniGPT-5, exhibits substantial improvement over the baseline Divter
model on the MMDialog dataset and consistently delivers superior or comparable
multimodal outputs in human evaluations on the VIST dataset, highlighting its
efficacy across diverse benchmarks.

分阶段训练方法和生成 vokens 提高大型语言模型在视觉和语言生成方面的效果。

MiniGPT-5：基于生成 Vokens 的交叉视觉与语言生成

MiniGPT-5: Interleaved Vision-and-Language Generation via Generative  Vokens

Due to the limitations of the model structure and pre-training objectives,
existing vision-and-language generation models cannot utilize pair-wise images
and text through bi-directional generation. In this paper, we propose DU-VLG, a
framework which unifies vision-and-language generation as sequence generation
problems. DU-VLG is trained with novel dual pre-training tasks: multi-modal
denoising autoencoder tasks and modality translation tasks. To bridge the gap
between image understanding and generation, we further design a novel
commitment loss. We compare pre-training objectives on image captioning and
text-to-image generation datasets. Results show that DU-VLG yields better
performance than variants trained with uni-directional generation objectives or
the variant without the commitment loss. We also obtain higher scores compared
to previous state-of-the-art systems on three vision-and-language generation
tasks. In addition, human judges further confirm that our model generates real
and relevant images as well as faithful and informative captions.

本论文提出了一种名为 DU-VLG 的框架，该框架将视觉和语言生成视为序列生成问题，并通过双向生成，利用对图像和文本的成对处理。采用多模态降噪自编码器任务和模态翻译任务进行双重预训练，并设计了一种新的承诺损失方法，以提高图像生成的质量。研究结果表明，与采用单向生成目标或不使用承诺损失的变体相比，DU-VLG 在图像字幕和文本到图像生成数据集上的性能更好，并在三个视觉和语言生成任务中获得了比以前的最先进系统更高的得分。此外，人类评测员进一步确认我们的模型生成了真实相关的图像并带有忠实和有信息的说明。