Mobile UI understanding is important for enabling various interaction tasks
such as UI automation and accessibility. Previous mobile UI modeling often
depends on the view hierarchy information of a screen, which directly provides
the structural data of the UI, with the hope to bypass challenging tasks of
visual modeling from screen pixels. However, view hierarchies are not always
available, and are often corrupted with missing object descriptions or
misaligned structure information. As a result, despite the use of view
hierarchies could offer short-term gains, it may ultimately hinder the
applicability and performance of the model. In this paper, we propose
Spotlight, a vision-only approach for mobile UI understanding. Specifically, we
enhance a vision-language model that only takes the screenshot of the UI and a
region of interest on the screen -- the focus -- as the input. This general
architecture of Spotlight is easily scalable and capable of performing a range
of UI modeling tasks. Our experiments show that our model establishes SoTA
results on several representative UI tasks and outperforms previous methods
that use both screenshots and view hierarchies as inputs. Furthermore, we
explore multi-task learning and few-shot prompting capacities of the proposed
models, demonstrating promising results in the multi-task learning direction.

本文提出了一种仅基于移动 UI 的截图进行 UI 建模的方法，命名为 Spotlight，该方法的表现优于使用截图和视图层次结构的方法，并且具有多任务学习和少样本学习能力。

移动界面理解：利用视觉 - 语言模型并聚焦

Spotlight: Mobile UI Understanding using Vision-Language Models with a  Focus

Supervised learning models often make systematic errors on rare subsets of
the data. When these subsets correspond to explicit labels in the data (e.g.,
gender, race) such poor performance can be identified straightforwardly. This
paper introduces a method for discovering systematic errors that do not
correspond to such explicitly labelled subgroups. The key idea is that similar
inputs tend to have similar representations in the final hidden layer of a
neural network. We leverage this structure by "shining a spotlight" on this
representation space to find contiguous regions where the model performs
poorly. We show that the spotlight surfaces semantically meaningful areas of
weakness in a wide variety of existing models spanning computer vision, NLP,
and recommender systems.

该文介绍了一种利用神经网络最终隐藏层相似输入具有相似表示的特性，寻找模型表现不良区域方法，旨在发现不符合显式标记子组的系统误差，并演示了该方法在计算机视觉、自然语言处理和推荐系统等多个现有模型中表现出的语义相关的弱点区域。