Recent generalist vision-language models (VLMs) have demonstrated impressive
reasoning capabilities across diverse multimodal tasks. However, these models
still struggle with fine-grained object-level understanding and grounding. In
terms of modeling, existing VLMs implicitly align text tokens with image patch
tokens, which is ineffective for embedding alignment at the same granularity
and inevitably introduces noisy spurious background features. Additionally,
these models struggle when generalizing to unseen visual concepts and may not
be reliable for domain-specific tasks without further fine-tuning. To address
these limitations, we propose a novel method to prompt large language models
with in-context visual object vectors, thereby enabling controllable
object-level reasoning. This eliminates the necessity of fusing a lengthy array
of image patch features and significantly speeds up training. Furthermore, we
propose region-level retrieval using our object representations, facilitating
rapid adaptation to new objects without additional training. Our experiments
reveal that our method achieves competitive referring object classification and
captioning performance, while also offering zero-shot generalization and
robustness to visually challenging contexts.

通过在上下文中引入视觉对象向量，我们提出了一种新的方法来引导大型语言模型，从而实现可控的对象级推理，消除了融合冗长图像区块特征的必要性，显著加速了训练。此外，我们还提出了使用对象表示进行区域级检索的方法，便于快速适应新对象而无需额外训练。我们的实验证明，我们的方法在参考对象分类和字幕生成性能上达到了竞争力，并提供了零样本泛化和对视觉上具有挑战性的情境的稳健性。