Representing the semantics of GUI screens and components is crucial to
data-driven computational methods for modeling user-GUI interactions and mining
GUI designs. Existing GUI semantic representations are limited to encoding
either the textual content, the visual design and layout patterns, or the app
contexts. Many representation techniques also require significant manual data
annotation efforts. This paper presents Screen2Vec, a new self-supervised
technique for generating representations in embedding vectors of GUI screens
and components that encode all of the above GUI features without requiring
manual annotation using the context of user interaction traces. Screen2Vec is
inspired by the word embedding method Word2Vec, but uses a new two-layer
pipeline informed by the structure of GUIs and interaction traces and
incorporates screen- and app-specific metadata. Through several sample
downstream tasks, we demonstrate Screen2Vec's key useful properties:
representing between-screen similarity through nearest neighbors,
composability, and capability to represent user tasks.

本文提出了 Screen2Vec 算法，该算法结合了 GUI 语义表示、嵌入向量和自监督的技术，在不需要手动注释的前提下，能够表示 GUI 屏幕和组件的所有特征，包括文字内容、视觉设计和布局模式、应用上下文，能够用于建模用户 - GUI 交互和挖掘 GUI 设计信息。