Last year, multimodal architectures served up a revolution in AI-based
approaches and solutions, extending the capabilities of large language models
(LLM). We propose an \textit{OmniFusion} model based on a pretrained LLM and
adapters for visual modality. We evaluated and compared several architecture
design principles for better text and visual data coupling: MLP and transformer
adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their
fusing approach, image encoding method (whole image or tiles encoding) and two
7B LLMs (the proprietary one and open-source Mistral). Experiments on 8
visual-language benchmarks show the top score for the best OmniFusion setup in
terms of different VQA tasks in comparison with open-source LLaVA-like
solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We
also propose a variety of situations, where OmniFusion provides highly-detailed
answers in different domains: housekeeping, sightseeing, culture, medicine,
handwritten and scanned equations recognition, etc. Mistral-based OmniFusion
model is an open-source solution with weights, training and inference scripts
available at this https URL

基于预训练的大型语言模型和视觉模态的适配器，我们提出了一种全融合模型 OmniFusion，通过比较多种架构设计原则、图像编码方法和不同的语言模型，在 8 个视觉语言基准测试中取得了最高得分，提供了在不同领域中的高度详细答案，同时我们还提供了 Mistral-based OmniFusion 模型的开源解决方案链接。