While some work attempt to generate front-end code intelligently from UI
screenshots, it may be more convenient to utilize UI design drafts in Sketch
which is a popular UI design software, because we can access multimodal UI
information directly such as layers type, position, size, and visual images.
However, fragmented layers could degrade the code quality without being merged
into a whole part if all of them are involved in the code generation. In this
paper, we propose a pipeline to merge fragmented layers automatically. We first
construct a graph representation for the layer tree of a UI draft and detect
all fragmented layers based on the visual features and graph neural networks.
Then a rule-based algorithm is designed to merge fragmented layers. Through
experiments on a newly constructed dataset, our approach can retrieve most
fragmented layers in UI design drafts, and achieve 87% accuracy in the
detection task, and the post-processing algorithm is developed to cluster
associative layers under simple and general circumstances.

本研究提出了一种自动合并 Sketch 中的分散图层的方法，通过构建 UI 设计草图中图层树的图形表示，基于视觉特征和图神经网络能够检测所有的分散图层，并通过基于规则的算法自动合并它们，能够在代码生成过程中提升代码质量和减少用户的工作量。