Natural language descriptions of user interface (UI) elements such as
alternative text are crucial for accessibility and language-based interaction
in general. Yet, these descriptions are constantly missing in mobile UIs. We
propose widget captioning, a novel task for automatically generating language
descriptions for UI elements from multimodal input including both the image and
the structural representations of user interfaces. We collected a large-scale
dataset for widget captioning with crowdsourcing. Our dataset contains 162,859
language phrases created by human workers for annotating 61,285 UI elements
across 21,750 unique UI screens. We thoroughly analyze the dataset, and train
and evaluate a set of deep model configurations to investigate how each feature
modality as well as the choice of learning strategies impact the quality of
predicted captions. The task formulation and the dataset as well as our
benchmark models contribute a solid basis for this novel multimodal captioning
task that connects language and user interfaces.

提出了一种通过多模态输入自动生成 UI 元素的语言描述的新任务：小部件字幕生成任务，并使用众包收集了一个大规模的包含 162,859 个语言短语的数据集以对此进行深入分析，该数据集涵盖了 21,750 个独特 UI 屏幕中的 61,285 个 UI 元素的结构表示和图像。