App developers use the Graphical User Interface (GUI) of other apps as an
important source of inspiration to design and improve their own apps. In recent
years, research suggested various approaches to retrieve GUI designs that fit a
certain text query from screenshot datasets acquired through automated GUI
exploration. However, such text-to-GUI retrieval approaches only leverage the
textual information of the GUI elements in the screenshots, neglecting visual
information such as icons or background images. In addition, the retrieved
screenshots are not steered by app developers and often lack important app
features, e.g. whose UI pages require user authentication. To overcome these
limitations, this paper proposes GUing, a GUI search engine based on a
vision-language model called UIClip, which we trained specifically for the app
GUI domain. For this, we first collected app introduction images from Google
Play, which usually display the most representative screenshots selected and
often captioned (i.e. labeled) by app vendors. Then, we developed an automated
pipeline to classify, crop, and extract the captions from these images. This
finally results in a large dataset which we share with this paper: including
303k app screenshots, out of which 135k have captions. We used this dataset to
train a novel vision-language model, which is, to the best of our knowledge,
the first of its kind in GUI retrieval. We evaluated our approach on various
datasets from related work and in manual experiment. The results demonstrate
that our model outperforms previous approaches in text-to-GUI retrieval
achieving a Recall@10 of up to 0.69 and a HIT@10 of 0.91. We also explored the
performance of UIClip for other GUI tasks including GUI classification and
Sketch-to-GUI retrieval with encouraging results.

本论文提出了一种基于名为 UIClip 的视觉语言模型的 GUI 搜索引擎 GUing，其通过训练特定于应用 GUI 领域的大型数据集，在文本查询中实现了优于以前方法的文本到 GUI 的检索。

GUing: 使用视觉语言模型的移动 GUI 搜索引擎

GUing: A Mobile GUI Search Engine using a Vision-Language Model

User interface (UI) design is a difficult yet important task for ensuring the
usability, accessibility, and aesthetic qualities of applications. In our
paper, we develop a machine-learned model, UIClip, for assessing the design
quality and visual relevance of a UI given its screenshot and natural language
description. To train UIClip, we used a combination of automated crawling,
synthetic augmentation, and human ratings to construct a large-scale dataset of
UIs, collated by description and ranked by design quality. Through training on
the dataset, UIClip implicitly learns properties of good and bad designs by i)
assigning a numerical score that represents a UI design's relevance and quality
and ii) providing design suggestions. In an evaluation that compared the
outputs of UIClip and other baselines to UIs rated by 12 human designers, we
found that UIClip achieved the highest agreement with ground-truth rankings.
Finally, we present three example applications that demonstrate how UIClip can
facilitate downstream applications that rely on instantaneous assessment of UI
design quality: i) UI code generation, ii) UI design tips generation, and iii)
quality-aware UI example search.

通过机器学习模型 UIClip，本研究开发了一种用于评估 UI 设计质量和视觉相关性的方法，通过截图和自然语言描述来对 UI 进行评估，实验证明 UIClip 在与其他基准方法和人工设计师评分进行对比时取得了最高一致性，从而为 UI 设计质量的直接评估提供了便利。