Recent object detection approaches rely on pretrained vision-language models
for image-text alignment. However, they fail to detect the Mobile User
Interface (MUI) element since it contains additional OCR information, which
describes its content and function but is often ignored. In this paper, we
develop a new MUI element detection dataset named MUI-zh and propose an
Adaptively Prompt Tuning (APT) module to take advantage of discriminating OCR
information. APT is a lightweight and effective module to jointly optimize
category prompts across different modalities. For every element, APT uniformly
encodes its visual features and OCR descriptions to dynamically adjust the
representation of frozen category prompts. We evaluate the effectiveness of our
plug-and-play APT upon several existing CLIP-based detectors for both standard
and open-vocabulary MUI element detection. Extensive experiments show that our
method achieves considerable improvements on two datasets. The datasets is
available at \url{github.com/antmachineintelligence/MUI-zh}.

本研究提出一种新的 MUI 元素检测数据集 MUI-zh，以及一种 Adaptive Prompt Tuning (APT) 模块，该模块可以通过采用区分 OCR 信息的方法，来联合优化不同模式下的类别 prompts。实验证明我们的方法可以取得显著的改善。

通过自适应提示调整进行移动用户界面元素检测

Mobile User Interface Element Detection Via Adaptively Prompt Tuning

Mobile User Interface Summarization generates succinct language descriptions
of mobile screens for conveying important contents and functionalities of the
screen, which can be useful for many language-based application scenarios. We
present Screen2Words, a novel screen summarization approach that automatically
encapsulates essential information of a UI screen into a coherent language
phrase. Summarizing mobile screens requires a holistic understanding of the
multi-modal data of mobile UIs, including text, image, structures as well as UI
semantics, motivating our multi-modal learning approach. We collected and
analyzed a large-scale screen summarization dataset annotated by human workers.
Our dataset contains more than 112k language summarization across $\sim$22k
unique UI screens. We then experimented with a set of deep models with
different configurations. Our evaluation of these models with both automatic
accuracy metrics and human rating shows that our approach can generate
high-quality summaries for mobile screens. We demonstrate potential use cases
of Screen2Words and open-source our dataset and model to lay the foundations
for further bridging language and user interfaces.

Screen2Words 是一种新颖的屏幕总结方法，它可以将 UI 屏幕的基本信息自动封装到一个连贯的语言短语中，并通过多模态学习，使用深度模型，为移动屏幕生成高质量的总结，为语言和用户界面的进一步桥梁奠定基础。

Screen2Words: 基于多模态学习的自动移动 UI 摘要

Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning

Searching for relative mobile user interface (UI) design examples can aid
interface designers in gaining inspiration and comparing design alternatives.
However, finding such design examples is challenging, especially as current
search systems rely on only text-based queries and do not consider the UI
structure and content into account. This paper introduces VINS, a visual search
framework, that takes as input a UI image (wireframe, high-fidelity) and
retrieves visually similar design examples. We first survey interface designers
to better understand their example finding process. We then develop a
large-scale UI dataset that provides an accurate specification of the
interface's view hierarchy (i.e., all the UI components and their specific
location). By utilizing this dataset, we propose an object-detection based
image retrieval framework that models the UI context and hierarchical
structure. The framework achieves a mean Average Precision of 76.39\% for the
UI detection and high performance in querying similar UI designs.

介绍了一种视觉搜索框架 VINS，它能够通过输入 UI 图像来检索类似的设计示例，并应用于移动用户界面的设计和体验研究。

VINS：移动用户界面设计的视觉搜索

VINS: Visual Search for Mobile User Interface Design

We present a new problem: grounding natural language instructions to mobile
user interface actions, and create three new datasets for it. For full task
evaluation, we create PIXELHELP, a corpus that pairs English instructions with
actions performed by people on a mobile UI emulator. To scale training, we
decouple the language and action data by (a) annotating action phrase spans in
HowTo instructions and (b) synthesizing grounded descriptions of actions for
mobile user interfaces. We use a Transformer to extract action phrase tuples
from long-range natural language instructions. A grounding Transformer then
contextually represents UI objects using both their content and screen position
and connects them to object descriptions. Given a starting screen and
instruction, our model achieves 70.59% accuracy on predicting complete
ground-truth action sequences in PIXELHELP.

本研究使用 Transformer 实现自然语言指令到移动用户界面动作的语义映射，并结合数据标注与合成技术，对其进行了完整的任务评估，并在 PIXELHELP 数据集上达到了 70.59% 的准确率。