Graphical User Interface (GUI) agents are designed to automate complex tasks
on digital devices, such as smartphones and desktops. Most existing GUI agents
interact with the environment through extracted structured data, which can be
notably lengthy (e.g., HTML) and occasionally inaccessible (e.g., on desktops).
To alleviate this issue, we propose a visual GUI agent -- SeeClick, which only
relies on screenshots for task automation. In our preliminary study, we have
discovered a key challenge in developing visual GUI agents: GUI grounding --
the capacity to accurately locate screen elements based on instructions. To
tackle this challenge, we propose to enhance SeeClick with GUI grounding
pre-training and devise a method to automate the curation of GUI grounding
data. Along with the efforts above, we have also created ScreenSpot, the first
realistic GUI grounding dataset that encompasses mobile, desktop, and web
environments. After pre-training, SeeClick demonstrates significant improvement
in ScreenSpot over various baselines. Moreover, comprehensive evaluations on
three widely used benchmarks consistently support our finding that advancements
in GUI grounding directly correlate with enhanced performance in downstream GUI
agent tasks. The model, data and code are available at
this https URL

通过仅依赖屏幕截图进行任务自动化，我们提出了一个视觉图形用户界面代理 - SeeClick，以解决开发图形用户界面代理的一个关键挑战：准确定位基于指令的屏幕元素。通过 GUI 定位的优化，我们观察到在各种基线测试中，SeeClick 在 ScreenSpot 上取得了显著提升，并且对三个广泛使用的基准测试进行的综合评估一致支持我们的发现：GUI 定位的进展与下游 GUI 代理任务的性能提升直接相关。