Most instance perception and image understanding solutions focus mainly on
natural images. However, applications for synthetic images, and more
specifically, images of Graphical User Interfaces (GUI) remain limited. This
hinders the development of autonomous computer-vision-powered Artificial
Intelligence (AI) agents. In this work, we present Search Instruction
Coordinates or SIC, a multi-modal solution for object identification in a GUI.
More precisely, given a natural language instruction and a screenshot of a GUI,
SIC locates the coordinates of the component on the screen where the
instruction would be executed. To this end, we develop two methods. The first
method is a three-part architecture that relies on a combination of a Large
Language Model (LLM) and an object detection model. The second approach uses a
multi-modal foundation model.

该研究提出了一种名为 Search Instruction Coordinates（SIC）的多模态对象识别解决方案，可用于自主计算机视觉驱动的人工智能代理，通过自然语言指令和 GUI 截图定位指令在屏幕上执行的组件的坐标。研究开发了两种方法，第一种基于大型语言模型和目标检测模型的三部分架构，第二种使用了多模态基础模型。

多模态人工智能系统中用于视觉指令建立的图形用户界面代理优化

Graphical user interface agents optimization for visual instruction  grounding using multi-modal artificial intelligence systems

All instance perception tasks aim at finding certain objects specified by
some queries such as category names, language expressions, and target
annotations, but this complete field has been split into multiple independent
subtasks. In this work, we present a universal instance perception model of the
next generation, termed UNINEXT. UNINEXT reformulates diverse instance
perception tasks into a unified object discovery and retrieval paradigm and can
flexibly perceive different types of objects by simply changing the input
prompts. This unified formulation brings the following benefits: (1) enormous
data from different tasks and label vocabularies can be exploited for jointly
training general instance-level representations, which is especially beneficial
for tasks lacking in training data. (2) the unified model is
parameter-efficient and can save redundant computation when handling multiple
tasks simultaneously. UNINEXT shows superior performance on 20 challenging
benchmarks from 10 instance-level tasks including classical image-level tasks
(object detection and instance segmentation), vision-and-language tasks
(referring expression comprehension and segmentation), and six video-level
object tracking tasks. Code is available at
this https URL

本文提出了一个称之为 UNINEXT 的通用实例感知模型，它将各种实例感知任务重新制定为统一的对象发现和检索范式，并且可以灵活地通过更改输入提示来感知不同类型的对象。这个统一的公式带来了以下好处：(1) 可以利用来自不同任务和标签词汇的大量数据进行联合训练，这对于缺乏训练数据的任务尤其有益。(2) 这个统一的模型是参数高效的，能够在处理多个任务时节省冗余计算。在 20 个具有挑战性的基准测试中，包括经典的图像级任务 (对象检测和实例分割)、视觉语言任务 (指称理解和分割) 和六个视频级对象跟踪任务，UNINEXT 表现出优异的性能。