Captioning images is a challenging scene-understanding task that connects
computer vision and natural language processing. While image captioning models
have been successful in producing excellent descriptions, the field has
primarily focused on generating a single sentence for 2D images. This paper
investigates whether integrating depth information with RGB images can enhance
the captioning task and generate better descriptions. For this purpose, we
propose a Transformer-based encoder-decoder framework for generating a
multi-sentence description of a 3D scene. The RGB image and its corresponding
depth map are provided as inputs to our framework, which combines them to
produce a better understanding of the input scene. Depth maps could be ground
truth or estimated, which makes our framework widely applicable to any RGB
captioning dataset. We explored different fusion approaches to fuse RGB and
depth images. The experiments are performed on the NYU-v2 dataset and the
Stanford image paragraph captioning dataset. During our work with the NYU-v2
dataset, we found inconsistent labeling that prevents the benefit of using
depth information to enhance the captioning task. The results were even worse
than using RGB images only. As a result, we propose a cleaned version of the
NYU-v2 dataset that is more consistent and informative. Our results on both
datasets demonstrate that the proposed framework effectively benefits from
depth information, whether it is ground truth or estimated, and generates
better captions. Code, pre-trained models, and the cleaned version of the
NYU-v2 dataset will be made publically available.

本研究通过整合 RGB 图像与深度信息进行图像描述，提出了一种基于 Transformer 编码 - 解码框架生成 3D 场景多句描述的方法，并在 NYU-v2 和斯坦福图像段落描述数据集上进行了实验，结果表明深度信息可以提高图像描述的效果。