We introduce Groma, a Multimodal Large Language Model (MLLM) with grounded
and fine-grained visual perception ability. Beyond holistic image
understanding, Groma is adept at region-level tasks such as region captioning
and visual grounding. Such capabilities are built upon a localized visual
tokenization mechanism, where an image input is decomposed into regions of
interest and subsequently encoded into region tokens. By integrating region
tokens into user instructions and model responses, we seamlessly enable Groma
to understand user-specified region inputs and ground its textual output to
images. Besides, to enhance the grounded chat ability of Groma, we curate a
visually grounded instruction dataset by leveraging the powerful GPT-4V and
visual prompting techniques. Compared with MLLMs that rely on the language
model or external module for localization, Groma consistently demonstrates
superior performances in standard referring and grounding benchmarks,
highlighting the advantages of embedding localization into image tokenization.
Project page: this https URL

Groma 是一个多模式大型语言模型，具有以图像感知为基础的细粒度视觉理解能力。它能够执行区域级任务并将图像与文字进行关联，通过在图像中定位兴趣区域并将其编码成区域标记的方式实现。此外，Groma 还利用 GPT-4V 和视觉提示技术创建了一个视觉基准数据集，使其在基准测试中表现出优越的对话能力。