Robotic vision applications often necessitate a wide range of visual
perception tasks, such as object detection, segmentation, and identification.
While there have been substantial advances in these individual tasks,
integrating specialized models into a unified vision pipeline presents
significant engineering challenges and costs. Recently, Multimodal Large
Language Models (MLLMs) have emerged as novel backbones for various downstream
tasks. We argue that leveraging the pre-training capabilities of MLLMs enables
the creation of a simplified framework, thus mitigating the need for
task-specific encoders. Specifically, the large-scale pretrained knowledge in
MLLMs allows for easier fine-tuning to downstream robotic vision tasks and
yields superior performance. We introduce the RoboLLM framework, equipped with
a BEiT-3 backbone, to address all visual perception tasks in the ARMBench
challenge-a large-scale robotic manipulation dataset about real-world warehouse
scenarios. RoboLLM not only outperforms existing baselines but also
substantially reduces the engineering burden associated with model selection
and tuning. The source code is publicly available at
this https URL

通过利用多模态大语言模型（MLLMs）的预训练能力，我们介绍了 RoboLLM 框架以应对 ARMBench 挑战中的视觉感知任务，在实际仓库场景下的大规模机器人操作数据集中，RoboLLM 不仅优于现有基线方法，还大幅减少了模型选择和调优的工作量。