Autonomous embodied agents live on an Internet of multimedia websites. Can
they hop around multimodal websites to complete complex user tasks? Existing
benchmarks fail to assess them in a realistic, evolving environment for their
embodiment across websites. To answer this question, we present MMInA, a
multihop and multimodal benchmark to evaluate the embodied agents for
compositional Internet tasks, with several appealing properties: 1) Evolving
real-world multimodal websites. Our benchmark uniquely operates on evolving
real-world websites, ensuring a high degree of realism and applicability to
natural user tasks. Our data includes 1,050 human-written tasks covering
various domains such as shopping and travel, with each task requiring the agent
to autonomously extract multimodal information from web pages as observations;
2) Multihop web browsing. Our dataset features naturally compositional tasks
that require information from or actions on multiple websites to solve, to
assess long-range reasoning capabilities on web tasks; 3) Holistic evaluation.
We propose a novel protocol for evaluating an agent's progress in completing
multihop tasks. We experiment with both standalone (multimodal) language models
and heuristic-based web agents. Extensive experiments demonstrate that while
long-chain multihop web tasks are easy for humans, they remain challenging for
state-of-the-art web agents. We identify that agents are more likely to fail on
the early hops when solving tasks of more hops, which results in lower task
success rates. To address this issue, we propose a simple memory augmentation
approach replaying past action trajectories to reflect. Our method
significantly improved both the single-hop and multihop web browsing abilities
of agents. See our code and data at this https URL

多模态网站的自主体代理在逐渐演变的真实环境中完成复杂用户任务具有挑战性，提出了 MMInA，这是一个多跳和多模态评估综合互联网任务能力的基准，通过构建真实世界的多模态网站和综合任务评估协议，发现自主体代理在长链多跳互联网任务方面存在挑战，提出了一种简单的记忆增强方法，明显提高了代理的单跳和多跳网络浏览能力。

MMInA：多跳多模态互联网代理的基准测试

MMInA: Benchmarking Multihop Multimodal Internet Agents

Following language instructions to navigate in unseen environments is a
challenging task for autonomous embodied agents. With strong representation
capabilities, pretrained vision-and-language models are widely used in VLN.
However, most of them are trained on web-crawled general-purpose datasets,
which incurs a considerable domain gap when used for VLN tasks. To address the
problem, we propose a novel and model-agnostic domain-aware prompt learning
(DAP) framework. For equipping the pretrained models with specific object-level
and scene-level cross-modal alignment in VLN tasks, DAP applies a low-cost
prompt tuning paradigm to learn soft visual prompts for extracting in-domain
image semantics. Specifically, we first generate a set of in-domain image-text
pairs with the help of the CLIP model. Then we introduce soft visual prompts in
the input space of the visual encoder in a pretrained model. DAP injects
in-domain visual knowledge into the visual encoder of the pretrained model in
an efficient way. Experimental results on both R2R and REVERIE show the
superiority of DAP compared to existing state-of-the-art methods.

通过引入低成本提示调整范式来学习特定领域视觉提示，我们提出了一种新颖且模型无关的领域感知提示学习（DAP）框架，以在 VLN 任务中为预训练模型提供特定对象级和场景级跨模态对齐，并将领域内视觉知识以高效的方式注入预训练模型的视觉编码器。在 R2R 和 REVERIE 上的实验结果显示，DAP 相比现有的最先进方法具有明显的优势。

DAP：面向领域感知的视觉与语言导航提示学习

DAP: Domain-aware Prompt Learning for Vision-and-Language Navigation

Implicit representations such as Neural Radiance Fields (NeRF) have been
shown to be very effective at novel view synthesis. However, these models
typically require manual and careful human data collection for training. In
this paper, we present AutoNeRF, a method to collect data required to train
NeRFs using autonomous embodied agents. Our method allows an agent to explore
an unseen environment efficiently and use the experience to build an implicit
map representation autonomously. We compare the impact of different exploration
strategies including handcrafted frontier-based exploration and modular
approaches composed of trained high-level planners and classical low-level path
followers. We train these models with different reward functions tailored to
this problem and evaluate the quality of the learned representations on four
different downstream tasks: classical viewpoint rendering, map reconstruction,
planning, and pose refinement. Empirical results show that NeRFs can be trained
on actively collected data using just a single episode of experience in an
unseen environment, and can be used for several downstream robotic tasks, and
that modular trained exploration models significantly outperform the classical
baselines.

该论文介绍了 AutoNeRF，一种使用自主实体代理收集数据以训练 NeRF 的方法，研究比较了不同的探索策略，经验证明可以使用主动收集的数据来训练 NeRF，并且可以用于下游机器人任务，以及模块化训练探索模型显著优于经典基线。