Zero-Shot Object Navigation (ZSON) enables agents to navigate towards
open-vocabulary objects in unknown environments. The existing works of ZSON
mainly focus on following individual instructions to find generic object
classes, neglecting the utilization of natural language interaction and the
complexities of identifying user-specific objects. To address these
limitations, we introduce Zero-shot Interactive Personalized Object Navigation
(ZIPON), where robots need to navigate to personalized goal objects while
engaging in conversations with users. To solve ZIPON, we propose a new
framework termed Open-woRld Interactive persOnalized Navigation (ORION), which
uses Large Language Models (LLMs) to make sequential decisions to manipulate
different modules for perception, navigation and communication. Experimental
results show that the performance of interactive agents that can leverage user
feedback exhibits significant improvement. However, obtaining a good balance
between task completion and the efficiency of navigation and interaction
remains challenging for all methods. We further provide more findings on the
impact of diverse user feedback forms on the agents' performance.

Zero-Shot Object Navigation (ZSON) mainly focuses on following individual instructions to find generic object classes, while Zero-shot Interactive Personalized Object Navigation (ZIPON) introduces Open-woRld Interactive persOnalized Navigation (ORION) framework that uses Large Language Models (LLMs) to navigate to personalized goal objects while engaging in conversations with users, showing significant improvement with interactive agents that can leverage user feedback, but the balance between task completion and the efficiency of navigation and interaction remains challenging.