Mobile UI understanding is important for enabling various interaction tasks
such as UI automation and accessibility. Previous mobile UI modeling often
depends on the view hierarchy information of a screen, which directly provides
the structural data of the UI, with the hope to bypass challenging tasks of
visual modeling from screen pixels. However, view hierarchies are not always
available, and are often corrupted with missing object descriptions or
misaligned structure information. As a result, despite the use of view
hierarchies could offer short-term gains, it may ultimately hinder the
applicability and performance of the model. In this paper, we propose
Spotlight, a vision-only approach for mobile UI understanding. Specifically, we
enhance a vision-language model that only takes the screenshot of the UI and a
region of interest on the screen -- the focus -- as the input. This general
architecture of Spotlight is easily scalable and capable of performing a range
of UI modeling tasks. Our experiments show that our model establishes SoTA
results on several representative UI tasks and outperforms previous methods
that use both screenshots and view hierarchies as inputs. Furthermore, we
explore multi-task learning and few-shot prompting capacities of the proposed
models, demonstrating promising results in the multi-task learning direction.

本文提出了一种仅基于移动 UI 的截图进行 UI 建模的方法，命名为 Spotlight，该方法的表现优于使用截图和视图层次结构的方法，并且具有多任务学习和少样本学习能力。

移动界面理解：利用视觉 - 语言模型并聚焦

Spotlight: Mobile UI Understanding using Vision-Language Models with a  Focus

We use a deep learning based approach to predict whether a selected element
in a mobile UI screenshot will be perceived by users as tappable, based on
pixels only instead of view hierarchies required by previous work. To help
designers better understand model predictions and to provide more actionable
design feedback than predictions alone, we additionally use ML interpretability
techniques to help explain the output of our model. We use XRAI to highlight
areas in the input screenshot that most strongly influence the tappability
prediction for the selected region, and use k-Nearest Neighbors to present the
most similar mobile UIs from the dataset with opposing influences on
tappability perception.

本文运用基于深度学习的方法，能够仅基于像素预测移动 UI 截屏中被用户感知为可点击的元素，并运用 ML 可解释性技术解释模型输出结果，其中采用 XRAI 技术突出显示对所选区域的可点击性预测影响最强烈的截屏区域，采用 k - 最近邻方法表示数据集中对可点击性感知产生相反影响的最相似移动 UI。

利用视觉建模和显著性分析预测和解释移动 UI 的可点击性

Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis

We present Magic Layouts; a method for parsing screenshots or hand-drawn
sketches of user interface (UI) layouts. Our core contribution is to extend
existing detectors to exploit a learned structural prior for UI designs,
enabling robust detection of UI components; buttons, text boxes and similar.
Specifically we learn a prior over mobile UI layouts, encoding common spatial
co-occurrence relationships between different UI components. Conditioning
region proposals using this prior leads to performance gains on UI layout
parsing for both hand-drawn UIs and app screenshots, which we demonstrate
within the context an interactive application for rapidly acquiring digital
prototypes of user experience (UX) designs.

本文介绍了 Magic Layouts 方法，该方法可以解析用户界面（UI）布局的截图或手绘草图。它在现有探测器的基础上扩展了一个结构先验，以便能够对 UI 组件进行强大的检测，并且使用该先验的条件区域提案可提高 UI 布局的解析性能。通过一个交互式应用程序，我们演示了该方法对手绘 UI 和应用截图的解析性能的提高，并用于快速获取用户体验（UX）设计的数字原型。