In this work, we propose a training-free method to inject visual referring into Multimodal Large Language Models (MLLMs) through learnable visual token optimization. We observe the relationship between text prompt tokens and visual tokens in MLLMs, where attention layers model the connection between them. Our approach involves adjusting visual tokens from the MLP output during inference, controlling which text prompt tokens attend to which visual tokens. We optimize a learnable visual token based on an energy function, enhancing the strength of referential regions in the attention map. This enables detailed region description and reasoning without the need for substantial training costs or model retraining. Our method offers a promising direction for integrating referential abilities into MLLMs. Our method support referring with box, mask, scribble and point. The results demonstrate that our method exhibits controllability and interpretability.

本研究解决了多模态大语言模型（MLLM）中视觉提示注入的无训练方法缺口。我们提出了一种通过可学习的视觉令牌优化来实现该目的的新颖方法，并在推理阶段优化视觉令牌，以增强注意力图中参考区域的强度。研究结果表明，该方法不仅具备良好的可控性和可解释性，还能有效整合参考能力。

ControlMLLM：无训练的多模态大语言模型视觉提示学习