Mobile UI understanding is important for enabling various interaction tasks
such as UI automation and accessibility. Previous mobile UI modeling often
depends on the view hierarchy information of a screen, which directly provides
the structural data of the UI, with the hope to bypass challenging tasks of
visual modeling from screen pixels. However, view hierarchies are not always
available, and are often corrupted with missing object descriptions or
misaligned structure information. As a result, despite the use of view
hierarchies could offer short-term gains, it may ultimately hinder the
applicability and performance of the model. In this paper, we propose
Spotlight, a vision-only approach for mobile UI understanding. Specifically, we
enhance a vision-language model that only takes the screenshot of the UI and a
region of interest on the screen -- the focus -- as the input. This general
architecture of Spotlight is easily scalable and capable of performing a range
of UI modeling tasks. Our experiments show that our model establishes SoTA
results on several representative UI tasks and outperforms previous methods
that use both screenshots and view hierarchies as inputs. Furthermore, we
explore multi-task learning and few-shot prompting capacities of the proposed
models, demonstrating promising results in the multi-task learning direction.

本文提出了一种仅基于移动 UI 的截图进行 UI 建模的方法，命名为 Spotlight，该方法的表现优于使用截图和视图层次结构的方法，并且具有多任务学习和少样本学习能力。