Multimodal Vision-Language Models (VLMs) enable powerful applications from
their fused understanding of images and language, but many perform poorly on UI
tasks due to the lack of UI training data. In this paper, we adapt a recipe for
generating paired text-image training data for VLMs to the UI domain by
combining existing pixel-based methods with a Large Language Model (LLM).
Unlike prior art, our method requires no human-provided annotations, and it can
be applied to any dataset of UI screenshots. We generate a dataset of 335K
conversational examples paired with UIs that cover Q&A, UI descriptions, and
planning, and use it to fine-tune a conversational VLM for UI tasks. To assess
the performance of our model, we benchmark it on UI element detection tasks,
evaluate response quality, and showcase its applicability to multi-step UI
navigation and planning.

使用生成配对文本 - 图像训练数据的方法，无需人工提供注释，适用于任何 UI 截图数据集，通过将现有的基于像素的方法与大型语言模型（LLM）相结合，对 UI 领域的 VLM 进行自动微调，以实现对 UI 任务的会话式 VLM 模型的性能评估、Q&A、UI 描述、规划以及多步 UI 导航和规划等方面的展示。