We introduce RealitySummary, a mixed reality reading assistant that can
enhance any printed or digital document using on-demand text extraction,
summarization, and augmentation. While augmented reading tools promise to
enhance physical reading experiences with overlaid digital content, prior
systems have typically required pre-processed documents, which limits their
generalizability and real-world use cases. In this paper, we explore on-demand
document augmentation by leveraging large language models. To understand
generalizable techniques for diverse documents, we first conducted an
exploratory design study which identified five categories of document
enhancements (summarization, augmentation, navigation, comparison, and
extraction). Based on this, we developed a proof-of-concept system that can
automatically extract and summarize text using Google Cloud OCR and GPT-4, then
embed information around documents using a Microsoft Hololens 2 and Apple
Vision Pro. We demonstrate real-time examples of six specific document
augmentations: 1) summaries, 2) comparison tables, 3) timelines, 4) keyword
lists, 5) summary highlighting, and 6) information cards. Results from a
usability study (N=12) and in-the-wild study (N=11) highlight the potential
benefits of on-demand MR document enhancement and opportunities for future
research.

我们介绍了一种名为 RealitySummary 的混合现实阅读助手，它可以通过按需的文本提取、摘要和增强来提升任何印刷或数字文档。我们探索了通过利用大规模语言模型进行按需文档增强的通用技术，并开发了一个概念验证系统，可以使用谷歌云 OCR 和 GPT-4 自动提取和摘要文本，然后使用 Microsoft Hololens 2 和 Apple Vision Pro 在文档周围嵌入信息。通过可用性研究和实地研究的结果，我们揭示了按需的混合现实文档增强的潜在效益和未来研究的机会。

RealitySummary：基于大型语言模型的按需混合现实文档增强

RealitySummary: On-Demand Mixed Reality Document Enhancement using Large  Language Models

Dense retrieval models, which aim at retrieving the most relevant document
for an input query on a dense representation space, have gained considerable
attention for their remarkable success. Yet, dense models require a vast amount
of labeled training data for notable performance, whereas it is often
challenging to acquire query-document pairs annotated by humans. To tackle this
problem, we propose a simple but effective Document Augmentation for dense
Retrieval (DAR) framework, which augments the representations of documents with
their interpolation and perturbation. We validate the performance of DAR on
retrieval tasks with two benchmark datasets, showing that the proposed DAR
significantly outperforms relevant baselines on the dense retrieval of both the
labeled and unlabeled documents.

提出一种 DAR 方案，通过对文档的内插和扰动来增强文档表示，实现无监督学习的稠密检索，显著优于相关基线。