User interface modeling is inherently multimodal, which involves several
distinct types of data: images, structures and language. The tasks are also
diverse, including object detection, language generation and grounding. In this
paper, we present VUT, a Versatile UI Transformer that takes multimodal input
and simultaneously accomplishes 5 distinct tasks with the same model. Our model
consists of a multimodal Transformer encoder that jointly encodes UI images and
structures, and performs UI object detection when the UI structures are absent
in the input. Our model also consists of an auto-regressive Transformer model
that encodes the language input and decodes output, for both question-answering
and command grounding with respect to the UI. Our experiments show that for
most of the tasks, when trained jointly for multi-tasks, VUT substantially
reduces the number of models and footprints needed for performing multiple
tasks, while achieving accuracy exceeding or on par with baseline models
trained for each individual task.

本研究提出了一种 VUT 多任务模型，可以同时完成 UI 镜像和结构编码、UI 对象检测、问题回答和命令接地等 5 个不同的任务。实验结果表明，VUT 在减少多任务模型数量和占用空间的同时，准确度也不逊于单独训练的基准模型。