Present-day graphical user interfaces (GUIs) exhibit diverse arrangements of
text, graphics, and interactive elements such as buttons and menus, but
representations of GUIs have not kept up. They do not encapsulate both semantic
and visuo-spatial relationships among elements. To seize machine learning's
potential for GUIs more efficiently, Graph4GUI exploits graph neural networks
to capture individual elements' properties and their semantic-visuo-spatial
constraints in a layout. The learned representation demonstrated its
effectiveness in multiple tasks, especially generating designs in a challenging
GUI autocompletion task, which involved predicting the positions of remaining
unplaced elements in a partially completed GUI. The new model's suggestions
showed alignment and visual appeal superior to the baseline method and received
higher subjective ratings for preference. Furthermore, we demonstrate the
practical benefits and efficiency advantages designers perceive when utilizing
our model as an autocompletion plug-in.

现代的图形用户界面（GUI）展示着不同的文本、图形和交互元素的排列，但 GUI 的表达方式却没有跟上。为了更高效地利用机器学习在 GUI 中的潜力，Graph4GUI 利用图神经网络捕捉了各个元素的属性以及它们之间的语义 - 视觉 - 空间约束关系。通过多项任务的效果验证，学习到的表示尤其在具有挑战性的 GUI 自动补全任务中表现出了其有效性，该任务涉及在部分完成的 GUI 中预测剩余未放置元素的位置。与基准方法相比，新模型的建议显示出更好的对齐和视觉吸引力，并且在主观评分上获得了更高的偏好度。此外，我们还展示了设计师在使用我们的模型作为自动补全插件时所感知到的实际效益和效率优势。

Graph4GUI：用于表示图形用户界面的图神经网络

Graph4GUI: Graph Neural Networks for Representing Graphical User  Interfaces

In the current landscape of pervasive smartphones and tablets, apps
frequently exist across both platforms. Although apps share most graphic user
interfaces (GUIs) and functionalities across phones and tablets, developers
often rebuild from scratch for tablet versions, escalating costs and
squandering existing design resources. Researchers are attempting to collect
data and employ deep learning in automated GUIs development to enhance
developers' productivity. There are currently several publicly accessible GUI
page datasets for phones, but none for pairwise GUIs between phones and
tablets. This poses a significant barrier to the employment of deep learning in
automated GUI development. In this paper, we introduce the Papt dataset, a
pioneering pairwise GUI dataset tailored for Android phones and tablets,
encompassing 10,035 phone-tablet GUI page pairs sourced from 5,593 unique app
pairs. We propose novel pairwise GUI collection approaches for constructing
this dataset and delineate its advantages over currently prevailing datasets in
the field. Through preliminary experiments on this dataset, we analyze the
present challenges of utilizing deep learning in automated GUI development.

在当前普及的智能手机和平板电脑的背景下，应用程序经常存在于两种平台上。尽管应用程序在手机和平板电脑上共享大部分图形用户界面（GUI）和功能，但开发人员通常需要从头开始重建平板电脑版本，导致成本升高，并浪费现有的设计资源。研究人员正在尝试收集数据并使用深度学习在自动 GUI 开发中提高开发人员的生产力。目前，有一些公开可访问的 GUI 页面数据集用于手机，但没有关于手机和平板电脑之间的配对 GUI 的数据集。这对于在自动 GUI 开发中采用深度学习构成了重大障碍。在本文中，我们介绍了 Papt 数据集，一个专门为 Android 手机和平板电脑定制的先驱性配对 GUI 数据集，包括来自 5593 个唯一应用程序对的 10035 个手机 - 平板电脑 GUI 页面对。我们提出了新颖的配对 GUI 收集方法以构建此数据集，并详细说明了它相对于当前流行数据集的优势。通过对此数据集的初步实验，我们分析了在自动 GUI 开发中利用深度学习所面临的挑战。

Android 手机和平板电脑间的双向 GUI 数据集构建

Pairwise GUI Dataset Construction Between Android Phones and Tablets

Much of the previous work towards digital agents for graphical user
interfaces (GUIs) has relied on text-based representations (derived from HTML
or other structured data sources), which are not always readily available.
These input representations have been often coupled with custom, task-specific
action spaces. This paper focuses on creating agents that interact with the
digital world using the same conceptual interface that humans commonly use --
via pixel-based screenshots and a generic action space corresponding to
keyboard and mouse actions. Building upon recent progress in pixel-based
pretraining, we show, for the first time, that it is possible for such agents
to outperform human crowdworkers on the MiniWob++ benchmark of GUI-based
instruction following tasks.

本研究提出一种使用基于像素的屏幕截图和一般的鼠标和键盘动作空间来模拟人与数字世界互动的智能代理。通过依赖于最近的像素预训练技术，该代理在 GUI 任务的 MinWob ++ 基准测试中表现出比人类众包工人更好的性能。