In this paper, we propose a new framework for online 3D scene perception. Conventional 3D scene perception methods are offline, i.e., take an already reconstructed 3D scene geometry as input, which is not applicable in robotic applications where the input data is streaming RGB-D videos rather than a complete 3D scene reconstructed from pre-collected RGB-D videos. To deal with online 3D scene perception tasks where data collection and perception should be performed simultaneously, the model should be able to process 3D scenes frame by frame and make use of the temporal information. To this end, we propose an adapter-based plug-and-play module for the backbone of 3D scene perception model, which constructs memory to cache and aggregate the extracted RGB-D features to empower offline models with temporal learning ability. Specifically, we propose a queued memory mechanism to cache the supporting point cloud and image features. Then we devise aggregation modules which directly perform on the memory and pass temporal information to current frame. We further propose 3D-to-2D adapter to enhance image features with strong global context. Our adapters can be easily inserted into mainstream offline architectures of different tasks and significantly boost their performance on online tasks. Extensive experiments on ScanNet and SceneNN datasets demonstrate our approach achieves leading performance on three 3D scene perception tasks compared with state-of-the-art online methods by simply finetuning existing offline models, without any model and task-specific designs. \href{https://xuxw98.github.io/Online3D/}{Project page}.

我们提出了一个新的在线三维场景感知框架，该框架可以在机器人应用中以流式的RGB-D视频作为输入数据，并使用时间信息对三维场景逐帧进行处理和感知。通过引入基于适配器的即插即用模块，我们构建了具有时间学习能力的离线模型的存储器，以缓存和聚合提取的RGB-D特征。在不需要任何模型和任务特定设计的情况下，我们的方法通过对现有离线模型进行微调，在ScanNet和SceneNN数据集上实验表明，在三个三维场景感知任务中相较于最先进的在线方法，我们的方法取得了领先的性能。

基于记忆的在线3D场景感知适配器