An essential topic for multimodal large language models (MLLMs) is aligning
vision and language concepts at a finer level. In particular, we devote efforts
to encoding visual referential information for tasks such as referring and
grounding. Existing methods, including proxy encoding and geometry encoding,
incorporate additional syntax to encode the object's location, bringing extra
burdens in training MLLMs to communicate between language and vision. This
study presents ClawMachine, offering a new methodology that notates an entity
directly using the visual tokens. It allows us to unify the prompt and answer
of visual referential tasks without additional syntax. Upon a joint
vision-language vocabulary, ClawMachine unifies visual referring and grounding
into an auto-regressive format and learns with a decoder-only architecture.
Experiments validate that our model achieves competitive performance across
visual referring and grounding tasks with a reduced demand for training data.
Additionally, ClawMachine demonstrates a native ability to integrate
multi-source information for complex visual reasoning, which prior MLLMs can
hardly perform without specific adaptions.

ClawMachine 是一种新的方法，通过直接使用视觉令牌来编码实体，统一了视觉引用和定位的自动回归格式，并通过仅有解码器的架构进行学习，实现了对视觉引用和定位任务的竞争性表现，并减少了对训练数据的需求。此外，ClawMachine 展示了在复杂视觉推理方面整合多源信息的本地能力，这是先前的多模态大型语言模型很难在没有特定适应的情况下实现的。