Humans can collaborate and complete tasks based on visual signals and
instruction from the environment. Training such a robot is difficult especially
due to the understanding of the instruction and the complicated environment.
Previous instruction-following agents are biased to English-centric corpus,
making it unrealizable to be applied to users that use multiple languages or
even low-resource languages. Nevertheless, the instruction-following agents are
pre-trained in a mode that assumes the user can observe the environment, which
limits its accessibility. In this work, we're trying to generalize the success
of instruction-following agents to non-English languages with little corpus
resources, and improve its intractability and accessibility. We introduce UVLN
(Universal Vision-Language Navigation), a novel machine-translation
instructional augmented framework for cross-lingual vision-language navigation,
with a novel composition of state-of-the-art large language model (GPT3) with
the image caption model (BLIP). We first collect a multilanguage
vision-language navigation dataset via machine translation. Then we extend the
standard VLN training objectives to a multilingual setting via a cross-lingual
language encoder. The alignment between different languages is captured through
a shared vision and action context via a cross-modal transformer, which encodes
the inputs of language instruction, visual observation, and action decision
sequences. To improve the intractability, we connect our agent with the large
language model that informs the situation and current state to the user and
also explains the action decisions. Experiments over Room Across Room Dataset
prove the effectiveness of our approach. And the qualitative results show the
promising intractability and accessibility of our instruction-following agent.

UVLN (Universal Vision-Language Navigation) 是一种新颖的增强型机器翻译指令框架，利用大型语言模型（GPT3）和图像标题模型（BLIP）的新颖组合，将传统的指令跟随代理推广到多语言和低资源语言等复杂领域，将不同语言之间的对齐通过跨模态变压器，对语言指令、视觉观察和动作决策序列进行编码、捕获和传递。

可访问的指令跟随代理

Accessible Instruction-Following Agent

We study the problem of continually training an instruction-following agent
through feedback provided by users during collaborative interactions. During
interaction, human users instruct an agent using natural language, and provide
realtime binary feedback as they observe the agent's instruction execution. We
cast learning as a contextual bandit problem, converting the user feedback to
immediate reward. We evaluate through multiple rounds of human-agent
interactions, demonstrating 15.4% absolute improvement in instruction execution
over time. We also show our approach is robust to several design variations,
and that the feedback signal is roughly equivalent to the learning signal of
supervised demonstration data.

通过人机协作交互提供的实时二元反馈，用自然语言训练指令遵从代理的问题被研究。将学习作为一种上下文医师问题，将用户反馈转换为立即奖励，证明了其在提高指令执行效果方面具有优势，并且反馈信号与监督式演示数据的学习信号基本等价。

基于实时反馈的指令跟随持续学习

Continual Learning for Instruction Following from Realtime Feedback

The ability to perform effective planning is crucial for building an
instruction-following agent. When navigating through a new environment, an
agent is challenged with (1) connecting the natural language instructions with
its progressively growing knowledge of the world; and (2) performing long-range
planning and decision making in the form of effective exploration and error
correction. Current methods are still limited on both fronts despite extensive
efforts. In this paper, we introduce the Evolving Graphical Planner (EGP), a
model that performs global planning for navigation based on raw sensory input.
The model dynamically constructs a graphical representation, generalizes the
action space to allow for more flexible decision making, and performs efficient
planning on a proxy graph representation. We evaluate our model on a
challenging Vision-and-Language Navigation (VLN) task with photorealistic
images and achieve superior performance compared to previous navigation
architectures. For instance, we achieve a 53% success rate on the test split of
the Room-to-Room navigation task through pure imitation learning, outperforming
previous navigation architectures by up to 5%.

本研究基于原始传感器输入，引入了进化图式计划者 (Evolving Graphical Planner，EGP) 模型，其能够以更灵活的决策空间，为导航执行全局规划， 并在具有照片级真实感的图像上的 Vision-and-Language Navigation (VLN) 任务中取得了优异的性能。