Recent advancements enlarge the capabilities of large language models (LLMs)
in zero-shot image-to-text generation and understanding by integrating
multi-modal inputs. However, such success is typically limited to English
scenarios due to the lack of large-scale and high-quality non-English
multi-modal resources, making it extremely difficult to establish competitive
counterparts in other languages. In this paper, we introduce the Ziya-VL
series, a set of bilingual large-scale vision-language models (LVLMs) designed
to incorporate visual semantics into LLM for multi-modal dialogue. Composed of
Ziya-VL-Base and Ziya-VL-Chat, our models adopt the Querying Transformer from
BLIP-2, further exploring the assistance of optimization schemes such as
instruction tuning, multi-stage training and low-rank adaptation module for
visual-language alignment. In addition, we stimulate the understanding ability
of GPT-4 in multi-modal scenarios, translating our gathered English image-text
datasets into Chinese and generating instruction-response through the
in-context learning method. The experiment results demonstrate that compared to
the existing LVLMs, Ziya-VL achieves competitive performance across a wide
range of English-only tasks including zero-shot image-text retrieval, image
captioning, and visual question answering. The evaluation leaderboard accessed
by GPT-4 also indicates that our models possess satisfactory image-text
understanding and generation capabilities in Chinese multi-modal scenario
dialogues. Code, demo and models are available at
~https://huggingface.co/IDEA-CCNL/Ziya-BLIP2-14B-Visual-v1.

通过引入视觉语义，将大规模的视觉 - 语言模型 (LVLMs) 融合到多模态对话中，Ziya-VL 在英语和汉语多模态场景中展现出了具有竞争力的图片 - 文本生成和理解能力。