Integrating RGB and NIR stereo imaging provides complementary spectral information, potentially enhancing robotic 3D vision in challenging lighting conditions. However, existing datasets and imaging systems lack pixel-level alignment between RGB and NIR images, posing challenges for downstream vision tasks. In this paper, we introduce a robotic vision system equipped with pixel-aligned RGB-NIR stereo cameras and a LiDAR sensor mounted on a mobile robot. The system simultaneously captures pixel-aligned pairs of RGB stereo images, NIR stereo images, and temporally synchronized LiDAR points. Utilizing the mobility of the robot, we present a dataset containing continuous video frames under diverse lighting conditions. We then introduce two methods that utilize the pixel-aligned RGB-NIR images: an RGB-NIR image fusion method and a feature fusion method. The first approach enables existing RGB-pretrained vision models to directly utilize RGB-NIR information without fine-tuning. The second approach fine-tunes existing vision models to more effectively utilize RGB-NIR information. Experimental results demonstrate the effectiveness of using pixel-aligned RGB-NIR images across diverse lighting conditions.

本研究解决了现有 RGB 和 NIR 图像在像素级别对齐不足的问题，影响机器人在复杂光照条件下的 3D 视觉能力。我们提出了一种新的机器人视觉系统，能够同时捕获像素对齐的 RGB 列队图像、NIR 列队图像以及时间同步的 LiDAR 点，并构建了一个在多种光照条件下的连续视频帧数据集。实验结果表明，利用像素对齐的 RGB-NIR 图像可显著提升视觉模型的表现。

像素对齐的 RGB-NIR 立体成像及其在机器人视觉中的数据集