Current universal segmentation methods demonstrate strong capabilities in
pixel-level image and video understanding. However, they lack reasoning
abilities and cannot be controlled via text instructions. In contrast, large
vision-language multimodal models exhibit powerful vision-based conversation
and reasoning capabilities but lack pixel-level understanding and have
difficulty accepting visual prompts for flexible user interaction. This paper
proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level
vision understanding with reasoning abilities. It can accept various visual and
text prompts for flexible user interaction. Specifically, we use a universal
segmentation method as the visual encoder, integrating image information,
perception priors, and visual prompts into visual tokens provided to the LLM.
The LLM is responsible for understanding the user's text instructions and
providing text responses and pixel-level segmentation results based on the
visual information. We propose perception prior embedding to better integrate
perception priors with image features. OMG-LLaVA achieves image-level,
object-level, and pixel-level reasoning and understanding in a single model,
matching or surpassing the performance of specialized methods on multiple
benchmarks. Rather than using LLM to connect each specialist, our work aims at
end-to-end training on one encoder, one decoder, and one LLM. The code and
model have been released for further research.

提出了一种新颖的 OMG-LLaVA 框架，将强大的像素级视觉理解与推理能力相结合，接受各种视觉和文本提示以实现灵活的用户交互。通过将视觉信息、感知先验和视觉提示整合为 LLM 可以理解用户的文本指令，并基于视觉信息提供文本响应和像素级分割结果。OMG-LLaVA 在一个模型中实现了图像级、对象级和像素级的推理和理解，达到了或超过了多个基准测试的专门方法的性能。