Most instance perception and image understanding solutions focus mainly on natural images. However, applications for synthetic images, and more specifically, images of Graphical User Interfaces (GUI) remain limited. This hinders the development of autonomous computer-vision-powered Artificial Intelligence (AI) agents. In this work, we present Search Instruction Coordinates or SIC, a multi-modal solution for object identification in a GUI. More precisely, given a natural language instruction and a screenshot of a GUI, SIC locates the coordinates of the component on the screen where the instruction would be executed. To this end, we develop two methods. The first method is a three-part architecture that relies on a combination of a Large Language Model (LLM) and an object detection model. The second approach uses a multi-modal foundation model.

该研究提出了一种名为Search Instruction Coordinates（SIC）的多模态对象识别解决方案，可用于自主计算机视觉驱动的人工智能代理，通过自然语言指令和GUI截图定位指令在屏幕上执行的组件的坐标。研究开发了两种方法，第一种基于大型语言模型和目标检测模型的三部分架构，第二种使用了多模态基础模型。

多模态人工智能系统中用于视觉指令建立的图形用户界面代理优化