Recent advancements in language and vision assistants have showcased
impressive capabilities but suffer from a lack of transparency, limiting
broader research and reproducibility. While open-source models handle general
image tasks effectively, they face challenges with the high computational
demands of complex visually-situated text understanding. Such tasks often
require increased token inputs and large vision modules to harness
high-resolution information. Striking a balance between model size and data
importance remains an open question. This study aims to redefine the design of
vision-language models by identifying key components and creating efficient
models with constrained inference costs. By strategically formulating datasets,
optimizing vision modules, and enhancing supervision techniques, we achieve
significant improvements in inference throughput while maintaining high
performance. Extensive experiments across models ranging from 160M to 13B
parameters offer insights into model optimization. We will fully open-source
our codebase, models, and datasets at this https URL .

通过识别关键组件和创建具有受限推理成本的高效模型，重新定义了视觉语言模型的设计，以实现推理吞吐量的显著提高，并保持高性能。