Detecting Graphical User Interface (GUI) elements in GUI images is a
domain-specific object detection task. It supports many software engineering
tasks, such as GUI animation and testing, GUI search and code generation.
Existing studies for GUI element detection directly borrow the mature methods
from computer vision (CV) domain, including old fashioned ones that rely on
traditional image processing features (e.g., canny edge, contours), and deep
learning models that learn to detect from large-scale GUI data. Unfortunately,
these CV methods are not originally designed with the awareness of the unique
characteristics of GUIs and GUI elements and the high localization accuracy of
the GUI element detection task. We conduct the first large-scale empirical
study of seven representative GUI element detection methods on over 50k GUI
images to understand the capabilities, limitations and effective designs of
these methods. This study not only sheds the light on the technical challenges
to be addressed but also informs the design of new GUI element detection
methods. We accordingly design a new GUI-specific old-fashioned method for
non-text GUI element detection which adopts a novel top-down coarse-to-fine
strategy, and incorporate it with the mature deep learning model for GUI text
detection.Our evaluation on 25,000 GUI images shows that our method
significantly advances the start-of-the-art performance in GUI element
detection.

本研究使用大型实证研究方法，比较七种 GUI 元素检测方法的性能和设计效果，设计一种新的基于旧有方法和深度学习模型的 GUI 元素检测方法，并在 25,000 张 GUI 图像上测试，结果表明我们的方法有着显著的改进。