In the realm of computer vision and robotics, embodied agents are expected to explore their environment and carry out human instructions. This necessitates the ability to fully understand 3D scenes given their first-person observations and contextualize them into language for interaction. However, traditional research focuses more on scene-level input and output setups from a global view. To address the gap, we introduce EmbodiedScan, a multi-modal, ego-centric 3D perception dataset and benchmark for holistic 3D scene understanding. It encompasses over 5k scans encapsulating 1M ego-centric RGB-D views, 1M language prompts, 160k 3D-oriented boxes spanning over 760 categories, some of which partially align with LVIS, and dense semantic occupancy with 80 common categories. Building upon this database, we introduce a baseline framework named Embodied Perceptron. It is capable of processing an arbitrary number of multi-modal inputs and demonstrates remarkable 3D perception capabilities, both within the two series of benchmarks we set up, i.e., fundamental 3D perception tasks and language-grounded tasks, and in the wild. Codes, datasets, and benchmarks will be available at https://github.com/OpenRobotLab/EmbodiedScan.

在计算机视觉和机器人领域，基于自身的智能体需要能够探索环境并执行人类指令，这要求其能够从第一人称观察中完全理解3D场景，并将其以语言的形式进行交互。本研究引入了一种名为EmbodiedScan的多模式、自我中心的3D感知数据集和基准，用于整体的3D场景理解。通过该数据集，我们还引入了一种名为Embodied Perceptron的基线框架，它能够处理任意数量的多模式输入，并展示出卓越的3D感知能力。

嵌入式扫描：面向身体智能的整体多模态3D感知套件