With the fast-growing GUI development workload in the Internet industry, some
work on intelligent methods attempted to generate maintainable front-end code
from UI screenshots. It can be more suitable for utilizing UI design drafts
that contain UI metadata. However, fragmented layers inevitably appear in the
UI design drafts which greatly reduces the quality of code generation. None of
the existing GUI automated techniques detects and merges the fragmented layers
to improve the accessibility of generated code. In this paper, we propose UI
Layers Merger (UILM), a vision-based method, which can automatically detect and
merge fragmented layers into UI components. Our UILM contains Merging Area
Detector (MAD) and a layers merging algorithm. MAD incorporates the boundary
prior knowledge to accurately detect the boundaries of UI components. Then, the
layers merging algorithm can search out the associated layers within the
components' boundaries and merge them into a whole part. We present a dynamic
data augmentation approach to boost the performance of MAD. We also construct a
large-scale UI dataset for training the MAD and testing the performance of
UILM. The experiment shows that the proposed method outperforms the best
baseline regarding merging area detection and achieves a decent accuracy
regarding layers merging.

本文提出了一个名为 UILM 的基于视觉的方法，它可以自动检测和合并碎片化的 UI 设计草稿中的碎片化层，以提高所生成代码的可访问性。实验证明，该方法在合并区域检测方面优于最佳基准线，并在图层合并方面实现了体面的精度。