Improving the accessibility and automation capabilities of mobile devices can
have a significant positive impact on the daily lives of countless users. To
stimulate research in this direction, we release a human-annotated dataset with
approximately 500k unique annotations aimed at increasing the understanding of
the functionality of UI elements. This dataset augments images and view
hierarchies from RICO, a large dataset of mobile UIs, with annotations for
icons based on their shapes and semantics, and associations between different
elements and their corresponding text labels, resulting in a significant
increase in the number of UI elements and the categories assigned to them. We
also release models using image-only and multimodal inputs; we experiment with
various architectures and study the benefits of using multimodal inputs on the
new dataset. Our models demonstrate strong performance on an evaluation set of
unseen apps, indicating their generalizability to newer screens. These models,
combined with the new dataset, can enable innovative functionalities like
referring to UI elements by their labels, improved coverage and better
semantics for icons etc., which would go a long way in making UIs more usable
for everyone.

通过对 RICO 数据集进行标注并使用多模态输入，该论文提出可提高移动设备的可访问性和自动化功能以便于使用的方法，使用户能够更好地了解 UI 元素的功能，并提出了一些创新性的功能，如通过标签引用 UI 元素，提高图标的语义等，以使 UI 对每个人都更加可用。

移动界面实现更优语义理解

Towards Better Semantic Understanding of Mobile Interfaces

Many accessibility features available on mobile platforms require
applications (apps) to provide complete and accurate metadata describing user
interface (UI) components. Unfortunately, many apps do not provide sufficient
metadata for accessibility features to work as expected. In this paper, we
explore inferring accessibility metadata for mobile apps from their pixels, as
the visual interfaces often best reflect an app's full functionality. We
trained a robust, fast, memory-efficient, on-device model to detect UI elements
using a dataset of 77,637 screens (from 4,068 iPhone apps) that we collected
and annotated. To further improve UI detections and add semantic information,
we introduced heuristics (e.g., UI grouping and ordering) and additional models
(e.g., recognize UI content, state, interactivity). We built Screen Recognition
to generate accessibility metadata to augment iOS VoiceOver. In a study with 9
screen reader users, we validated that our approach improves the accessibility
of existing mobile apps, enabling even previously inaccessible apps to be used.

本文介绍使用屏幕识别的方法来推断移动应用程序的辅助功能元数据，以改善移动应用程序的可访问性，从而使以前无法使用的应用程序能够被使用。