Vision-and-Language Navigation (VLN) aims to develop embodied agents that
navigate based on human instructions. However, current VLN frameworks often
rely on static environments and optimal expert supervision, limiting their
real-world applicability. To address this, we introduce Human-Aware
Vision-and-Language Navigation (HA-VLN), extending traditional VLN by
incorporating dynamic human activities and relaxing key assumptions. We propose
the Human-Aware 3D (HA3D) simulator, which combines dynamic human activities
with the Matterport3D dataset, and the Human-Aware Room-to-Room (HA-R2R)
dataset, extending R2R with human activity descriptions. To tackle HA-VLN
challenges, we present the Expert-Supervised Cross-Modal (VLN-CM) and
Non-Expert-Supervised Decision Transformer (VLN-DT) agents, utilizing
cross-modal fusion and diverse training strategies for effective navigation in
dynamic human environments. A comprehensive evaluation, including metrics
considering human activities, and systematic analysis of HA-VLN's unique
challenges, underscores the need for further research to enhance HA-VLN agents'
real-world robustness and adaptability. Ultimately, this work provides
benchmarks and insights for future research on embodied AI and Sim2Real
transfer, paving the way for more realistic and applicable VLN systems in
human-populated environments.

Vision-and-Language Navigation involves developing embodied agents that navigate based on human instructions. This paper introduces Human-Aware Vision-and-Language Navigation by incorporating dynamic human activities and relaxing key assumptions, utilizing the HA3D simulator, HA-R2R dataset, Expert-Supervised Cross-Modal, and Non-Expert-Supervised Decision Transformer agents for effective navigation in dynamic human environments.