Computer vision was long a tool used for aiding visually impaired people to
move around their environment and avoid obstacles and falls. Solutions are
limited to either indoor or outdoor scenes, which limits the kind of places and
scenes visually disabled people can be in, including entertainment places such
as theatres. Furthermore, most of the proposed computer-vision-based methods
rely on RGB benchmarks to train their models resulting in a limited performance
due to the absence of the depth modality.
In this paper, we propose a novel RGB-D dataset containing theatre scenes
with ground truth human actions and dense captions annotations for image
captioning and human action recognition: TS-RGBD dataset. It includes three
types of data: RGB, depth, and skeleton sequences, captured by Microsoft
Kinect.
We test image captioning models on our dataset as well as some skeleton-based
human action recognition models in order to extend the range of environment
types where a visually disabled person can be, by detecting human actions and
textually describing appearances of regions of interest in theatre scenes.

在这篇论文中，我们提出了一个包含具有人类动作的真实场景和密集标题注释的新型 RGB-D 数据集（TS-RGBD 数据集），以进行影像字幕和人类动作识别。我们测试了图像字幕模型和基于骨骼的人类动作识别模型，以通过检测人类动作并对剧场场景中感兴趣区域的外观进行文本描述，拓展了视障人士所在环境类型的范围。