This paper proposes an interactive navigation framework by using large
language and vision-language models, allowing robots to navigate in
environments with traversable obstacles. We utilize the large language model
(GPT-3.5) and the open-set Vision-language Model (Grounding DINO) to create an
action-aware costmap to perform effective path planning without fine-tuning.
With the large models, we can achieve an end-to-end system from textual
instructions like "Can you pass through the curtains to deliver medicines to
me?", to bounding boxes (e.g., curtains) with action-aware attributes. They can
be used to segment LiDAR point clouds into two parts: traversable and
untraversable parts, and then an action-aware costmap is constructed for
generating a feasible path. The pre-trained large models have great
generalization ability and do not require additional annotated data for
training, allowing fast deployment in the interactive navigation tasks. We
choose to use multiple traversable objects such as curtains and grasses for
verification by instructing the robot to traverse them. Besides, traversing
curtains in a medical scenario was tested. All experimental results
demonstrated the proposed framework's effectiveness and adaptability to diverse
environments.

提出一个互动导航框架，使用大型语言和视觉语言模型，使机器人能在存在可通过的障碍物的环境中导航。利用大型语言模型（GPT-3.5）和开放域视觉语言模型（Grounding DINO），创建一个动作感知的成本图，可以实现有效的路径规划而无需微调。实验结果证明了该框架在不同环境中的有效性和适应性。