With the surge in the development of large language models, embodied
intelligence has attracted increasing attention. Nevertheless, prior works on
embodied intelligence typically encode scene or historical memory in an
unimodal manner, either visual or linguistic, which complicates the alignment
of the model's action planning with embodied control. To overcome this
limitation, we introduce the Multimodal Embodied Interactive Agent (MEIA),
capable of translating high-level tasks expressed in natural language into a
sequence of executable actions. Specifically, we propose a novel Multimodal
Environment Memory (MEM) module, facilitating the integration of embodied
control with large models through the visual-language memory of scenes. This
capability enables MEIA to generate executable action plans based on diverse
requirements and the robot's capabilities. We conduct experiments in a dynamic
virtual cafe environment, utilizing multiple large models through zero-shot
learning, and carefully design scenarios for various situations. The
experimental results showcase the promising performance of our MEIA in various
embodied interactive tasks.

通过多模态环境记忆模块，我们提出了多模态交互式智能体（MEIA），能够将自然语言表达的高级任务转化为可执行动作序列，从而实现了大型模型与具有体现性控制的集成，实验结果展示了 MEIA 在各种交互任务中的良好表现。

咖啡厅场景的多模态体验互动代理

Multimodal Embodied Interactive Agent for Cafe Scene

Embodied control requires agents to leverage multi-modal pre-training to
quickly learn how to act in new environments, where video demonstrations
contain visual and motion details needed for low-level perception and control,
and language instructions support generalization with abstract, symbolic
structures. While recent approaches apply contrastive learning to force
alignment between the two modalities, we hypothesize better modeling their
complementary differences can lead to more holistic representations for
downstream adaption. To this end, we propose Emergent Communication for
Embodied Control (EC^2), a novel scheme to pre-train video-language
representations for few-shot embodied control. The key idea is to learn an
unsupervised "language" of videos via emergent communication, which bridges the
semantics of video details and structures of natural language. We learn
embodied representations of video trajectories, emergent language, and natural
language using a language model, which is then used to finetune a lightweight
policy network for downstream control. Through extensive experiments in
Metaworld and Franka Kitchen embodied benchmarks, EC^2 is shown to consistently
outperform previous contrastive learning methods for both videos and texts as
task inputs. Further ablations confirm the importance of the emergent language,
which is beneficial for both video and language learning, and significantly
superior to using pre-trained video captions. We also present a quantitative
and qualitative analysis of the emergent language and discuss future directions
toward better understanding and leveraging emergent communication in embodied
tasks.

提出了一种名为 EC^2 的新颖方案，用于预训练视频语言表示以进行 few-shot embodied control，通过紧密结合视频细节的语义和自然语言结构的紧密联系，实现了代理在新环境中快速行动的目标。该方法在 Metaworld 和 Franka Kitchen 的实验基准中表现了显着的优势，超过了以前的对比学习方法的表现。此外，对提出的联合建模方法进行了定量和定性分析，并讨论了未来的研究方向。

EC^2: 基于体现控制的紧急通讯

EC^2: Emergent Communication for Embodied Control

Parallel developments in neuroscience and deep learning have led to mutually
productive exchanges, pushing our understanding of real and artificial neural
networks in sensory and cognitive systems. However, this interaction between
fields is less developed in the study of motor control. In this work, we
develop a virtual rodent as a platform for the grounded study of motor activity
in artificial models of embodied control. We then use this platform to study
motor activity across contexts by training a model to solve four complex tasks.
Using methods familiar to neuroscientists, we describe the behavioral
representations and algorithms employed by different layers of the network
using a neuroethological approach to characterize motor activity relative to
the rodent's behavior and goals. We find that the model uses two classes of
representations which respectively encode the task-specific behavioral
strategies and task-invariant behavioral kinematics. These representations are
reflected in the sequential activity and population dynamics of neural
subpopulations. Overall, the virtual rodent facilitates grounded collaborations
between deep reinforcement learning and motor neuroscience.

本文介绍了一种基于虚拟老鼠的平台，用于研究人工模型的运动活动，研究表明该模型使用两类表示方法分别编码特定于任务的行为策略和任务不变的行为运动学，以及描述了使用神经习性学方法研究不同层次的神经网络在行为观察和目标下的算法，并指出虚拟老鼠便于深度强化学习和运动神经科学之间的合作。