This paper presents PaLI-3, a smaller, faster, and stronger vision language
model (VLM) that compares favorably to similar models that are 10x larger. As
part of arriving at this strong performance, we compare Vision Transformer
(ViT) models pretrained using classification objectives to contrastively
(SigLIP) pretrained ones. We find that, while slightly underperforming on
standard image classification benchmarks, SigLIP-based PaLI shows superior
performance across various multimodal benchmarks, especially on localization
and visually-situated text understanding. We scale the SigLIP image encoder up
to 2 billion parameters, and achieves a new state-of-the-art on multilingual
cross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindles
research on fundamental pieces of complex VLMs, and could fuel a new generation
of scaled-up models.

PaLI-3 是一个较小、更快、更强的视觉语言模型（VLM），相比于大十倍的类似模型具有有利的比较性能。与使用分类目标进行预训练的 Vision Transformer（ViT）模型相比，SigLIP 预训练模型的 PaLI 在标准图像分类基准上略有弱势，但在各种多模态基准测试中表现出优越的性能，特别是在定位和视觉文本理解方面。我们将 SigLIP 图像编码器扩展到 20 亿参数，并在多语言交叉模态检索方面取得了新的最先进水平。我们希望 PaLI-3 能够重新点燃对复杂 VLM 基础要素的研究，并推动新一代扩展模型的发展。