Automated software testing is integral to the software development process,
streamlining workflows and ensuring product reliability. Visual testing within
this context, especially concerning user interface (UI) and user experience
(UX) validation, stands as one of crucial determinants of overall software
quality. Nevertheless, conventional methods like pixel-wise comparison and
region-based visual change detection fall short in capturing contextual
similarities, nuanced alterations, and understanding the spatial relationships
between UI elements. In this paper, we introduce a novel graph-based method for
visual change detection in software test automation. Leveraging a machine
learning model, our method accurately identifies UI controls from software
screenshots and constructs a graph representing contextual and spatial
relationships between the controls. This information is then used to find
correspondence between UI controls within screenshots of different versions of
a software. The resulting graph encapsulates the intricate layout of the UI and
underlying contextual relations, providing a holistic and context-aware model.
This model is finally used to detect and highlight visual regressions in the
UI. Comprehensive experiments on different datasets showed that our change
detector can accurately detect visual software changes in various simple and
complex test scenarios. Moreover, it outperformed pixel-wise comparison and
region-based baselines by a large margin in more complex testing scenarios.
This work not only contributes to the advancement of visual change detection
but also holds practical implications, offering a robust solution for
real-world software test automation challenges, enhancing reliability, and
ensuring the seamless evolution of software interfaces.

这项研究介绍了一种基于图形的方法，用于软件测试自动化中的可视变化检测，通过机器学习模型准确识别软件截屏中的用户界面控件，并构建表示控件之间上下文和空间关系的图形，利用该模型可以检测和突出显示用户界面的视觉回归，该方法在各种简单和复杂测试场景中准确检测出软件的视觉变化，在更复杂的测试场景中明显优于像素级比较和基于区域的基准方法，这项工作不仅推动了可视变化检测的发展，还具有实际意义，提供了一个强大的解决方案，用于解决现实世界的软件测试自动化挑战，提高可靠性，确保软件界面的无缝演进。