Engaging in recreational activities in public spaces poses challenges for
blind people, often involving dependency on sighted help. Window shopping is a
key recreational activity that remains inaccessible. In this paper, we
investigate the information needs, challenges, and current approaches blind
people have to recreational window shopping to inform the design of existing
wayfinding and navigation technology for supporting blind shoppers in
exploration and serendipitous discovery. We conduct a formative study with a
total of 18 blind participants that include both focus groups (N=8) and
interviews for requirements analysis (N=10). We find that there is a desire for
push notifications of promotional information and pull notifications about
shops of interest such as the targeted audience of a brand. Information about
obstacles and points-of-interest required customization depending on one's
mobility aid as well as presence of a crowd, children, and wheelchair users. We
translate these findings into specific information modalities and rendering in
the context of two existing AI-infused assistive applications: NavCog (a
turn-by-turn navigation app) and Cabot (a navigation robot).

对视觉障碍者参与公共空间的娱乐活动存在依赖视力帮助的挑战，其中包括无法参与的购物活动。本文通过研究盲人对娱乐购物的信息需求、挑战和现有方法，旨在为设计现有的导航技术提供支持，帮助盲人购物者进行探索和偶然发现。通过对 18 名盲人参与者进行形成性研究，包括焦点小组和需求分析访谈，我们发现盲人希望获得促销信息的推送通知，以及关于感兴趣店铺的拉取通知，例如品牌的目标受众。关于障碍物和兴趣点的信息需要根据个人的移动辅助设备以及人群、儿童和轮椅使用者的存在进行自定义。我们将这些发现应用于两个现有的 AI 辅助应用程序（NavCog 和 Cabot），具体包括信息模态和渲染方面。

借助 AI 技术支持盲人在娱乐性窗口购物中

"We are at the mercy of others' opinion": Supporting Blind People in  Recreational Window Shopping with AI-infused Technology

One of the ways blind people understand their surroundings is by clicking
images and relying on descriptions generated by image captioning systems.
Current work on captioning images for the visually impaired do not use the
textual data present in the image when generating captions. This problem is
critical as many visual scenes contain text. Moreover, up to 21% of the
questions asked by blind people about the images they click pertain to the text
present in them. In this work, we propose altering AoANet, a state-of-the-art
image captioning model, to leverage the text detected in the image as an input
feature. In addition, we use a pointer-generator mechanism to copy the detected
text to the caption when tokens need to be reproduced accurately. Our model
outperforms AoANet on the benchmark dataset VizWiz, giving a 35% and 16.2%
performance improvement on CIDEr and SPICE scores, respectively.

本研究提出用 AoANet 和指针 - 生成机制来改进图像字幕视频，实现文本检测和复制，提高图像字幕性能从而更好地帮助视力受损人士理解环境。

多模态图像字幕生成（面向视觉障碍人士）

Multi-Modal Image Captioning for the Visually Impaired

The study of algorithms to automatically answer visual questions currently is
motivated by visual question answering (VQA) datasets constructed in artificial
VQA settings. We propose VizWiz, the first goal-oriented VQA dataset arising
from a natural VQA setting. VizWiz consists of over 31,000 visual questions
originating from blind people who each took a picture using a mobile phone and
recorded a spoken question about it, together with 10 crowdsourced answers per
visual question. VizWiz differs from the many existing VQA datasets because (1)
images are captured by blind photographers and so are often poor quality, (2)
questions are spoken and so are more conversational, and (3) often visual
questions cannot be answered. Evaluation of modern algorithms for answering
visual questions and deciding if a visual question is answerable reveals that
VizWiz is a challenging dataset. We introduce this dataset to encourage a
larger community to develop more generalized algorithms that can assist blind
people.

该研究提出了第一个目标导向 VQA 数据集 VizWiz，通过 blind people 所拍摄的图像和口述的问题构建数据集，旨在鼓励更多的社区开发更广泛的算法帮助盲人进行视觉问题的回答和解决。