Large Language Models (LLMs) have garnered significant attention for their
advancements in natural language processing, demonstrating unparalleled prowess
in text comprehension and generation. Yet, the simultaneous generation of
images with coherent textual narratives remains an evolving frontier. In
response, we introduce an innovative interleaved vision-and-language generation
technique anchored by the concept of "generative vokens," acting as the bridge
for harmonized image-text outputs. Our approach is characterized by a
distinctive two-staged training strategy focusing on description-free
multimodal generation, where the training requires no comprehensive
descriptions of images. To bolster model integrity, classifier-free guidance is
incorporated, enhancing the effectiveness of vokens on image generation. Our
model, MiniGPT-5, exhibits substantial improvement over the baseline Divter
model on the MMDialog dataset and consistently delivers superior or comparable
multimodal outputs in human evaluations on the VIST dataset, highlighting its
efficacy across diverse benchmarks.

分阶段训练方法和生成 vokens 提高大型语言模型在视觉和语言生成方面的效果。