We train a suite of multimodal foundation models (MMFM) using the popular LLaVA framework with the recently released Gemma family of large language models (LLMs). Of particular interest is the 2B parameter Gemma model, which provides opportunities to construct capable small-scale MMFMs. In line with findings from other papers in this space, we test the effect of ablating three design features: pretraining the connector, utilizing a more powerful image backbone, and increasing the size of the language backbone. The resulting models, which we call LLaVA-Gemma, exhibit moderate performance on an array of evaluations, but fail to improve past the current comparably sized SOTA models. Closer analysis of performance shows mixed effects; skipping pretraining tends to reduce performance, larger vision models sometimes improve performance, and increasing language model size has inconsistent effects. We publicly release training recipes, code and weights for our models for the LLaVA-Gemma models.

我们使用近期发布的Gemma系列大型语言模型（LLMs）在流行的LLaVA框架中训练了一套多模态基础模型（MMFM）。我们测试了删除三个设计特征对模型性能的影响，并对性能进行了深入分析。最终的LLaVA-Gemma模型在多项评估中表现适中，但未能超越当前规模相当的SOTA模型。

LLaVA-Gemma：利用紧凑的语言模型加速多模基础模型