User Interface (UI) understanding has been an increasingly popular topic over
the last few years. So far, there has been a vast focus solely on web and
mobile applications. In this paper, we introduce the harder task of computer UI
understanding. With the goal of enabling research in this field, we have
generated a dataset with a set of videos where a user is performing a sequence
of actions and each image shows the desktop contents at that time point. We
also present a framework that is composed of a synthetic sample generation
pipeline to augment the dataset with relevant characteristics, and a
contrastive learning method to classify images in the videos. We take advantage
of the natural conditional, tree-like, relationship of the images'
characteristics to regularize the learning of the representations by dealing
with multiple partial tasks simultaneously. Experimental results show that the
proposed framework outperforms previously proposed hierarchical multi-label
contrastive losses in fine-grain UI classification.

介绍了计算机用户界面 (UI) 理解的研究课题，包括创造一个视频数据集、合成样本生成管道和对图像进行对比学习的框架，实验证明该框架优于先前提出的层级多标签对比损失在细粒度用户界面分类中的表现。