App developers use the Graphical User Interface (GUI) of other apps as an
important source of inspiration to design and improve their own apps. In recent
years, research suggested various approaches to retrieve GUI designs that fit a
certain text query from screenshot datasets acquired through automated GUI
exploration. However, such text-to-GUI retrieval approaches only leverage the
textual information of the GUI elements in the screenshots, neglecting visual
information such as icons or background images. In addition, the retrieved
screenshots are not steered by app developers and often lack important app
features, e.g. whose UI pages require user authentication. To overcome these
limitations, this paper proposes GUing, a GUI search engine based on a
vision-language model called UIClip, which we trained specifically for the app
GUI domain. For this, we first collected app introduction images from Google
Play, which usually display the most representative screenshots selected and
often captioned (i.e. labeled) by app vendors. Then, we developed an automated
pipeline to classify, crop, and extract the captions from these images. This
finally results in a large dataset which we share with this paper: including
303k app screenshots, out of which 135k have captions. We used this dataset to
train a novel vision-language model, which is, to the best of our knowledge,
the first of its kind in GUI retrieval. We evaluated our approach on various
datasets from related work and in manual experiment. The results demonstrate
that our model outperforms previous approaches in text-to-GUI retrieval
achieving a Recall@10 of up to 0.69 and a HIT@10 of 0.91. We also explored the
performance of UIClip for other GUI tasks including GUI classification and
Sketch-to-GUI retrieval with encouraging results.

本论文提出了一种基于名为 UIClip 的视觉语言模型的 GUI 搜索引擎 GUing，其通过训练特定于应用 GUI 领域的大型数据集，在文本查询中实现了优于以前方法的文本到 GUI 的检索。