Large Vision Language Models (LVLMs) have demonstrated impressive zero-shot
capabilities in various vision-language dialogue scenarios. However, the
absence of fine-grained visual object detection hinders the model from
understanding the details of images, leading to irreparable visual
hallucinations and factual errors. In this paper, we propose Lyrics, a novel
multi-modal pre-training and instruction fine-tuning paradigm that bootstraps
vision-language alignment from fine-grained cross-modal collaboration. Building
on the foundation of BLIP-2, Lyrics infuses local visual features extracted
from a visual refiner that includes image tagging, object detection and
semantic segmentation modules into the Querying Transformer, while on the text
side, the language inputs equip the boundary boxes and tags derived from the
visual refiner. We further introduce a two-stage training scheme, in which the
pre-training stage bridges the modality gap through explicit and comprehensive
vision-language alignment targets. During the instruction fine-tuning stage, we
introduce semantic-aware visual feature extraction, a crucial method that
enables the model to extract informative features from concrete visual objects.
Our approach achieves strong performance on 13 held-out datasets across various
vision-language tasks, and demonstrates promising multi-modal understanding and
detailed depiction capabilities in real dialogue scenarios.

一种新的多模式预训练和指令微调范式 Lyrics，通过细粒度的跨模态协作，将从细粒度视觉精炼器中提取的局部视觉特征注入到查询转换器中，同时，在语言输入的边界框和标签方面使用从视觉精炼器中推导出的信息，通过两阶段训练方案实现模态融合，从而在各种视觉语言任务中取得了强大的性能和有希望的多模态理解和详细描绘能力。