Humans can learn to operate the user interface (UI) of an application by
reading an instruction manual or how-to guide. Along with text, these resources
include visual content such as UI screenshots and images of application icons
referenced in the text. We explore how to leverage this data to learn generic
visio-linguistic representations of UI screens and their components. These
representations are useful in many real applications, such as accessibility,
voice navigation, and task automation. Prior UI representation models rely on
UI metadata (UI trees and accessibility labels), which is often missing,
incompletely defined, or not accessible. We avoid such a dependency, and
propose Lexi, a pre-trained vision and language model designed to handle the
unique features of UI screens, including their text richness and context
sensitivity. To train Lexi we curate the UICaption dataset consisting of 114k
UI images paired with descriptions of their functionality. We evaluate Lexi on
four tasks: UI action entailment, instruction-based UI image retrieval,
grounding referring expressions, and UI entity recognition.

本研究探索如何利用 UI 说明书中的图像来学习 UI 屏幕及其组件的通用视觉 - 语言表示，并提出了一种名为 Lexi 的预训练模型，该模型可处理 UI 屏幕的独特特点，包括其丰富的文本和上下文敏感性。通过创建包含 114k UI 图像及其功能描述的 UICaption 数据集，我们证明了该模型可用于 UI 行动蕴含和 UI 实体识别等任务。