Human-AI interactivity is a critical aspect that reflects the usability of
multimodal large language models (MLLMs). However, existing end-to-end MLLMs
only allow users to interact with them through language instructions, leading
to the limitation of the interactive accuracy and efficiency. In this study, we
present precise referring instructions that utilize diverse reference
representations such as points and boxes as referring prompts to refer to the
special region. This enables MLLMs to focus on the region of interest and
achieve finer-grained interaction. Based on precise referring instruction, we
propose ChatSpot, a unified end-to-end multimodal large language model that
supports diverse forms of interactivity including mouse clicks, drag-and-drop,
and drawing boxes, which provides a more flexible and seamless interactive
experience. We also construct a multi-grained vision-language
instruction-following dataset based on existing datasets and GPT-4 generating.
Furthermore, we design a series of evaluation tasks to assess the effectiveness
of region recognition and interaction. Experimental results showcase ChatSpot's
promising performance.

基于精确的指代指令，我们提出了 ChatSpot，这是一个统一的端到端多模态大型语言模型，支持多种形式的交互，包括鼠标点击、拖放和绘制矩形框，从而提供更灵活、无缝的交互体验。实验证明 ChatSpot 具有良好的性能。