We introduce RealitySummary, a mixed reality reading assistant that can
enhance any printed or digital document using on-demand text extraction,
summarization, and augmentation. While augmented reading tools promise to
enhance physical reading experiences with overlaid digital content, prior
systems have typically required pre-processed documents, which limits their
generalizability and real-world use cases. In this paper, we explore on-demand
document augmentation by leveraging large language models. To understand
generalizable techniques for diverse documents, we first conducted an
exploratory design study which identified five categories of document
enhancements (summarization, augmentation, navigation, comparison, and
extraction). Based on this, we developed a proof-of-concept system that can
automatically extract and summarize text using Google Cloud OCR and GPT-4, then
embed information around documents using a Microsoft Hololens 2 and Apple
Vision Pro. We demonstrate real-time examples of six specific document
augmentations: 1) summaries, 2) comparison tables, 3) timelines, 4) keyword
lists, 5) summary highlighting, and 6) information cards. Results from a
usability study (N=12) and in-the-wild study (N=11) highlight the potential
benefits of on-demand MR document enhancement and opportunities for future
research.

我们介绍了一种名为 RealitySummary 的混合现实阅读助手，它可以通过按需的文本提取、摘要和增强来提升任何印刷或数字文档。我们探索了通过利用大规模语言模型进行按需文档增强的通用技术，并开发了一个概念验证系统，可以使用谷歌云 OCR 和 GPT-4 自动提取和摘要文本，然后使用 Microsoft Hololens 2 和 Apple Vision Pro 在文档周围嵌入信息。通过可用性研究和实地研究的结果，我们揭示了按需的混合现实文档增强的潜在效益和未来研究的机会。