We train a suite of multimodal foundation models (MMFM) using the popular
LLaVA framework with the recently released Gemma family of large language
models (LLMs). Of particular interest is the 2B parameter Gemma model, which
provides opportunities to construct capable small-scale MMFMs. In line with
findings from other papers in this space, we test the effect of ablating three
design features: pretraining the connector, utilizing a more powerful image
backbone, and increasing the size of the language backbone. The resulting
models, which we call LLaVA-Gemma, exhibit moderate performance on an array of
evaluations, but fail to improve past the current comparably sized SOTA models.
Closer analysis of performance shows mixed effects; skipping pretraining tends
to reduce performance, larger vision models sometimes improve performance, and
increasing language model size has inconsistent effects. We publicly release
training recipes, code and weights for our models for the LLaVA-Gemma models.

我们使用近期发布的 Gemma 系列大型语言模型（LLMs）在流行的 LLaVA 框架中训练了一套多模态基础模型（MMFM）。我们测试了删除三个设计特征对模型性能的影响，并对性能进行了深入分析。最终的 LLaVA-Gemma 模型在多项评估中表现适中，但未能超越当前规模相当的 SOTA 模型。