Graphical User Interface (or simply UI) is a primary mean of interaction
between users and their device. In this paper, we discuss three major
complementary approaches on how to use Artificial Intelligence (AI) to support
app designers create better, more diverse, and creative UI of mobile apps.
First, designers can prompt a Large Language Model (LLM) like GPT to directly
generate and adjust one or multiple UIs. Second, a Vision-Language Model (VLM)
enables designers to effectively search a large screenshot dataset, e.g. from
apps published in app stores. The third approach is to train a Diffusion Model
(DM) specifically designed to generate app UIs as inspirational images. We
discuss how AI should be used, in general, to inspire and assist creative app
design rather than automating it.

通过本文讨论了三种不同的方法来利用人工智能 (AI) 来支持应用设计师创造更好、更多样化和更具创意的移动应用界面 (UI)：第一种方法是设计师可以使用大型语言模型 (LLM)，如 GPT，直接生成和调整一个或多个 UI；第二种方法是使用视觉 - 语言模型 (VLM) 有效地搜索大型截图数据集，例如应用商店中发布的应用；第三种方法是训练一个专门设计用于生成应用界面的扩散模型 (DM)，作为启发性图片。我们讨论了如何使用 AI 来激发和辅助创造性应用设计，而不是自动化它。

关于人工智能启发的用户界面设计

On AI-Inspired UI-Design

Prototyping AI applications is notoriously difficult. While large language
model (LLM) prompting has dramatically lowered the barriers to AI prototyping,
designers are still prototyping AI functionality and UI separately. We
investigate how coupling prompt and UI design affects designers' workflows.
Grounding this research, we developed PromptInfuser, a Figma plugin that
enables users to create semi-functional mockups, by connecting UI elements to
the inputs and outputs of prompts. In a study with 14 designers, we compare
PromptInfuser to designers' current AI-prototyping workflow. PromptInfuser was
perceived to be significantly more useful for communicating product ideas, more
capable of producing prototypes that realistically represent the envisioned
artifact, more efficient for prototyping, and more helpful for anticipating UI
issues and technical constraints. PromptInfuser encouraged iteration over
prompt and UI together, which helped designers identify UI and prompt
incompatibilities and reflect upon their total solution. Together, these
findings inform future systems for prototyping AI applications.

通过比较 PromptInfuser 和设计师当前的 AI 原型化工作流程，我们研究了联结提示和 UI 设计如何影响设计师的工作流程，发现 PromptInfuser 比当前工作流程更有用，更能产生逼真的原型，更高效，更有帮助于预测 UI 问题和技术限制，从而为设计 AI 应用的原型系统提供了参考。

PromptInfuser: 紧密耦合人工智能与 UI 设计如何影响设计师的工作流程

PromptInfuser: How Tightly Coupling AI and UI Design Impacts Designers'  Workflows

While some work attempt to generate front-end code intelligently from UI
screenshots, it may be more convenient to utilize UI design drafts in Sketch
which is a popular UI design software, because we can access multimodal UI
information directly such as layers type, position, size, and visual images.
However, fragmented layers could degrade the code quality without being merged
into a whole part if all of them are involved in the code generation. In this
paper, we propose a pipeline to merge fragmented layers automatically. We first
construct a graph representation for the layer tree of a UI draft and detect
all fragmented layers based on the visual features and graph neural networks.
Then a rule-based algorithm is designed to merge fragmented layers. Through
experiments on a newly constructed dataset, our approach can retrieve most
fragmented layers in UI design drafts, and achieve 87% accuracy in the
detection task, and the post-processing algorithm is developed to cluster
associative layers under simple and general circumstances.

本研究提出了一种自动合并 Sketch 中的分散图层的方法，通过构建 UI 设计草图中图层树的图形表示，基于视觉特征和图神经网络能够检测所有的分散图层，并通过基于规则的算法自动合并它们，能够在代码生成过程中提升代码质量和减少用户的工作量。